id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
24240	Setting Content-Encoding on a non-ASCII StreamingHttpResponse raises UnicodeEncodeError	Matthew Somerville	nobody	"HttpResponseBase's make_bytes() assumes that if the Content-Encoding header is set, then everything has already been dealt with bytes-wise, but in a streaming situation this is not necessarily the case as the header has been set before make_bytes() will be called on the original content.

For example, if you provide an iterable including a non-ASCII Unicode string to a StreamingHttpResponse and are using the GZip middleware, the response will die with a UnicodeEncodeError when it arrives at the first non-ASCII Unicode chunk.

I have added two tests at https://github.com/dracos/django/compare/non-ascii-compressed-streaming (let me know if you'd prefer a PR), firstly a working one to show that supplying non-ASCII Unicode strings to a StreamingHttpResponse works fine normally; then a failing one to show that passing such a response through the GZip middleware dies with a UnicodeEncodeError.

This appears to be happening because as we're working with an iterable, make_bytes() is only called when necessary, but by the point that happens and the iterable is consumed, the middleware has added the Content-Encoding header and thus make_bytes() tries to call bytes(value) (and dies). If it had been a normal HttpResponse, make_bytes() would have been called when the content was set, well before the middleware set the Content-Encoding header.

I'm not aware of other intricacies of HttpResponse to suggest an obvious fix, sorry. The start of this short circuit was back in #4969 but lots has changed since then; I don't know if this special case when Content-Encoding is set is still necessary in the same way."	Bug	closed	HTTP handling	1.7	Normal	fixed			Ready for checkin	1	0	0	0	0	0