Make Request proxy the WSGI environ

[simon]

Mark Ramm proposed this at DjangoCon. Filing it here for further discussion.

[mramm]

The idea here is that there should be easy access to a wsgi environ like dictionary, to make calling wsgi applications from within django trivial.

I would even propose that making a call_wsgi_app function that grabs the environ, creates a start_response callable, calls the wsgi app, and returns the response properly.

This would let users write code like:

def someview(request, *args)

do_something(args)
return call_wsgi_app(wsgi_app, request)

This would allow people who use django with SQLAlchemy to user RUM (A django admin like wsgi app) from a veiw, or to use some SOAP wsgi application, or a TurboGears? app, or whatever they want in a Django view.

[Graham Dumpleton]

Not that it helps with the general case, but when using Apache/mod_wsgi it is quite easy to overlay different WSGI applications, with code for each executing in same interpreter context. This can be done without modifying Django. Doing it the Apache/mod_wsgi way though doesn't give you access to any Django request object, but if the intent is to host an independent WSGI application within same URL namespace, then you don't need that. Note though that you would still be able to access the Django internals and frameworks if the point of the WSGI application is to still be hooking into Django internals or database layer in some way.

To get this working with Apache/mod_wsgi you just need to ensure that WSGIApplicationGroup is set to be same value for Django and the separate WSGI application.

Alias /media/ /usr/local/django/mysite/media/

<Directory /usr/local/django/mysite/media>
Order deny,allow
Allow from all
</Directory>

# Force all hosted applications to run in same interpreter.
WSGIApplicationGroup applications

WSGIScriptAlias / /usr/local/django/mysite/apache/django.wsgi

<Directory /usr/local/django/mysite/apache>
Order deny,allow
Allow from all
</Directory>

WSGIScriptAlias /appl /usr/local/wsgi-applications/app.wsgi

<Directory /usr/local/wsgi-applications>
Order deny,allow
Allow from all
</Directory>

Similar thing can be done in mod_python as long as you have a mod_python/WSGI bridge installed. In case of mod_python, since default is that virtual host uses one interpreter, rather than each application, you may not even have to use PythonInterpreter.

For most other hosting mechanisms you can still obviously overlay applications under same virtual host, but each couldn't be made to readily run in the same process and interpreter. This may not be the end of things though as the distinct process the WSGI application is running in could still load Django to manipulate things, just like a cron job can.

All up, being able to embed a WSGI application within Django only makes a lot of sense if the response from the WSGI application is going to be passed through some sort of Django output filtering mechanism for doing stuff, or if request details and content are similarly morphed on the way into the WSGI application. That or the WSGI application is not really a distinct WSGI application but is more of a hybrid thing which is dependent upon being able to access Django per request information and work on that. In that case, why use WSGI when one could just write it to Django APIs to begin with.

So, as much as academically this may be an interesting thing to do, most hosting mechanisms allow you to overlay multiple applications at different URLs. Thus, I'd suggest there really needs to be a compelling reason for this, else it is just extra work you probably don't need.

[Graham Dumpleton]

FWIW, one valid reason suggested over in discussion on mod_wsgi list about this has been the need to easily wrap a separate WSGI application within Django session based login mechanism.

[Dirkjan Ochtman]

One other thing where this came up is with hosting a Mercurial application (push/pull interface) within Django (that is, same url-space, same authentication/authorization). That kind of thing would be massively easier if Django provided some bits to do this kind of thing.

[(none)]

Milestone post-1.0 deleted

[Ryan Kelly]

Thought I'd throw in my 2c now that this is on the radar for a GSoC project...

I have a WSGI application that accepts file uploads as standard multipart/form-data, and I want to expose it from within a larger Django application. This is complicated by the fact that I need to trigger both Django's authentication framework *and* Django's file upload handlers, to validate and monitor the uploads before they hit the WSGI app.

My current solution is pretty yuck and duplicates a lot of logic from multipartparser.py, but it's been working well for me so far. The idea is to set environ['wsgi.input'] to a wrapper object, which knows about Django's POST and FILE dictionaries and can convert these to/from a standard readable input stream as required by WSGI. It's on github if you want to have a look, adapted from some work by Travis Parker:

​http://github.com/rfk/django/blob/f96eb6068552e228f705beda9c8181db2079a797/django/wsgi/wsgi_to_django.py

This kind of thing might be far more complicated than you're willing to support in Django, but is worth thinking about at least.

Cheers,

Ryan

[Gustavo Narea]

Hi.

I need to get this fixed so we can use Repoze software in the Django app we maintain at work. I already started working on it and wanted to you know so we won't duplicate efforts.

Please feel free to assign this ticket to me if you want - I'll propose a patch once I'm done anyway.

Cheers,

• Gustavo.

[Gustavo Narea]

Patch to make the WSGI available where it's missing

[Gustavo Narea]

Please let me know what you think about the patch. I found that request.environ was usually defined, but was missing in mod_python, so I just made it available everywhere no matter the handler being used.

[Carl Meyer]

It's become a bit muddy what would qualify as a "fix" for this bug. request.environ exists; the only patch that's been submitted here is to make it available under mod_python as well, but that's not really relevant since the mod_python handler is deprecated and soon to be removed. Having requests "proxy" the environ suggests perhaps something deeper, maybe that changes to the HttpRequest instance would actually be reflected in corresponding changes in the environ? Not sure how important that actually is in practice.

Looking at Mark Ramm's initial comment about call_wsgi_app, it seems that the actual request is to include something like ​django-wsgi in core, to allow easy "adaptation" between Django view functions and/or urlconfs and WSGI callables. The fact that that code is so short demonstrates that the needed pieces are already in place, it's a matter of making it more convenient.

I think adding such adapter functions to Django core makes sense, and fills in really the last major gap in Django WSGI support. Filed also as ​https://github.com/alex/django-wsgi/issues/2

[Carl Meyer]

Just realized that the WSGI/view/urlconf adapter stuff is already covered by #12091.

Given that, and that WSGIRequest (used by WSGIHandler) already makes the WSGI environ available via its environ attribute, and once ModPythonHandler is removed WSGIHandler will be the only request handler left in Django, I'm marking this bug fixed.

Please reopen if "proxy" is intended to mean something more than just making the WSGI environ available (and clarify why exactly something more would be necessary).

[Gustavo Narea]

"Proxy" means that if, for example, you set "request.path = '/foo'", the change will propagate to "request.environPATH_INFO".

[Carl Meyer]

Replying to Gustavo:

"Proxy" means that if, for example, you set "request.path = '/foo'", the change will propagate to "request.environPATH_INFO".

I see. There does not appear to be a consensus within the WSGI community (even outside Django) that this is a good idea. WebOb does this, Werkzeug does not. I just discussed it on IRC with mitsuhiko, and there are some real issues with implementing this. Like - what all do you attempt to proxy? Even in "simple" cases like the path, you have to deal with encoding back and forth, since request.path is unicode and request.environ['PATH_INFO'] is a string. What about the request body (form data)? To proxy this back, you're kind of fighting against WSGI - it sounds like WebOb dumps it into a temporary file, which is complicated and possibly slow.

I understand that if you have a mutable request object, not proxying changes back to the WSGI environ can cause inconsistencies if you later dispatch internally to another WSGI app. Werkzeug deals with this by documenting the request object as immutable (though in fact it isn't entirely). We do already have people in Django modifying request.path in middleware (e.g. django-localeurl does this to make the locale URL prefix transparent to the app), so it's a bit late for us to take the "request is immutable" approach.

Personally, I'm not convinced this is necessary for good WSGI support. I think we could just document that changes to the request are not reflected in the WSGI environ, and people in the (unusual) situation of both modifying the request object and dispatching to another WSGI app can deal with it for their specific situation.

Another option, if the only real use case for this is when you dispatch to another WSGI app, would be to implement some of this in the call_wsgi_app wrapper or whatever implements #12091. In other words, changes to the request would not immediately be reflected in request.environ, but call_wsgi_app would attempt to notice some cases (like request.path modification) and reflect those changes in the WSGI environ it passes on to the called app. This would feel less invasive to me, and seems like it would meet the use case adequately.

[Aymeric Augustin]

Django's WSGI support has improved a lot since this ticket was filed, and as explained by Carl in comments 15 and 18, I would be hard to retrofit this feature. It's a fairly large backwards incompatibility for debatable benefits.

If you feel strongly about this, please make a precise proposal on the django-developers mailing list, addressing backwards-compatibility concerns.