File-like interface for HttpRequest

[Ivan Sagalaev]

This is an implementation of a file-like interface for HttpRequest. Primary goal is to allow iterative reading and handling of raw request data without loading it into memory entirely. Secondary goal is refactoring some code duplication in ModPythonRequest and WSGIRequest.

[Ivan Sagalaev]

Patch for review

[Ivan Sagalaev]

Attached first patch for review. No changes to docs and tests yet, just code. Here's an overview of what and why has changed:

• Both subclasses of HttpRequest "publish" their internal byte stream to self._stream which is then accessed from HttpRequest
• Both subclasses also define a boolean self._read_started that is set later to True from HttpRequest methods to signal that reading from the stream has started and things like raw_post_data and POST aren't acessible
• On the other hand, if raw_post_data is accessed before any attempt to read from the stream it's wrapped in StringIO and becomes request._stream itself.
• Parsing of post data by upload handlers and reading of raw_post_data is extracted into HttpRequest because they were almost identical in both subclasses. I liked very much a comment in one of the copies of _load_post_and_files saying "read the proper comment in another file" :-)
• core.handlers.wsgi now has a special wrapper class LimitedInput which takes a stream, content_length as its size limit and defines read and readline for it. Important thing is that it's used only when working with development server. It is needed because dev server passes as 'wsgi.input' a pure socket._fileobject which hangs and waits forever when one tries to read past data that it has in it. I decided not to wrap proper production streams from mod_python and wsgi servers into LimitedInput to be able to use their own implementations of read and readline that I suspect to be more efficient. Both mod_python and flup are smart enough to close streams when read completely.

Comments are welcome!

[Ivan Sagalaev]

Just tested with Apache + mod_wsgi. Works good.

[Ivan Sagalaev]

Jacob, Malcolm, please clarify does "DDN" mean that we should discuss it in django-developers and if yes, does it make sense to do it after or before 1.1?

[Jacob]

I marked this DDN because I want a second opinion before this gets accepted; bringing it to django-dev is one way for you to prod that process along.

However, since this is a feature addition it would need to be in the beta to get into 1.1, and there's a long list of other features we decided upon earlier in the release process. So realistically if this gets in it'll be 1.2.

[Ivan Sagalaev]

New patch with docs and tests

[Ivan Sagalaev]

Patch updated to current trunk

[Ivan Sagalaev]

An answer to Russel's concern about it playing well with WSGI improvement branch: ​http://groups.google.com/group/django-developers/msg/a0716eff23949427

I've looked at the branch's code and am certain that we don't conflict. That branch is changing response part of HTTP handling only and this patch is about refactoring request part. That simple :-)

[Ivan Sagalaev]

Patch updated to current trunk

[Ivan Sagalaev]

Updated to current trunk, tests converted to unittests

[Ivan Sagalaev]

Patch updated to current trunk

[Russell Keith-Magee]

(In [14394]) Fixed #9886 -- Added a file-like interface to HttpRequest. Thanks to Ivan Sagalaev for the suggestion and patch.