id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
7581	Middleware accessing HttpResponse.content breaks streaming HttpResponse objects.	Tai Lee	ccahoon	"{{{ConditionalGetMiddleware}}}, {{{GZipMiddleware}}}, {{{CommonMiddleware}}}, and {{{CsrfMiddleware}}} all access response.content directly or indirectly. This prevents a streaming response from being initiated until any generator passed to the {{{HttpResponse}}} as content is consumed, which can cause a timeout when trying to stream large dynamically generated content.

I've already put together a patch based on the assumption that {{{HttpResponse._is_string}}} being {{{False}}} indicates content has been passed in as a dynamic generator and thus we shouldn't delay streaming a response to the browser while consuming the entire generator.

The patch implements the following:

* Allow middleware to assign a new generator to {{{HttpResponse.content}}} without consuming it (e.g. {{{GZipMiddleware}}})

* Compress chunks of content yielded by {{{HttpResponse._container}}} progressively in {{{GZipMiddleware}}} to allow streaming GZipped content

* Only generate an ETag in {{{CommonMiddleware}}} from {{{HttpResponse.content}}} if {{{HttpResponse._is_string}}} is {{{True}}}

* Only check that the length of {{{HttpResponse.content}}} is less than 200 in {{{GZipMiddleware}}} if {{{HttpResponse._is_string}}} is {{{True}}}

* Only set the Content-Length header in {{{GZipMiddleware}}} and {{{ConditionalGetMiddleware}}} if {{{HttpResponse._is_string}}} is {{{True}}}

With {{{CommonMiddleware}}} enabled by default and breaking the streaming response functionality if ETags are enabled in {{{settings.py}}}, I'd consider this a bug. It can be worked around by manually specifying a bogus ETag before returning the response, which doesn't seem ideal.

With this patch, users still have the option of consuming a generator before passing it to the {{{HttpResponse}}} in order to enable ETag and Content-Length headers, and conditional GZipping when the content length is less than 200.

With {{{CsrfMiddleware}}}, the generator is only consumed when the Content-Type header is {{{text/html}}} or {{{application/xhtml+xml}}}, which may be an acceptable compromise - no streaming response for HTML content if you choose to use {{{django.contrib.csrf}}}.

This patch accesses {{{HttpResponse._is_string}}} and {{{HttpResponse._container}}} from external middleware classes. Perhaps these properties could be made public and/or renamed to be more logical in this context?

"	New feature	closed	Core (Other)	dev	Normal	fixed	stream HttpResponse Content-Length	real.human@… mmitar@… mindsocket	Accepted	1	0	0	0	0	0