id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
29323	HTTPRequest QueryDict wrongly decodes binary encoded values	Thomas Riccardi	nobody	"`application/x-www-form-urlencoded` bodies allow binary values: just percent-escape (most of) the bytes.

It doesn't work with python2 and django.

In python2, QueryDict returns an `unicode` string for values, which makes it impossible to represent binary data.
Worse: it returns an `unicode` string whose unicode code points are the raw expected bytes (caused by lax behavior of `str.decode('iso-8859-1')`) so it silently fails.

Example:
{{{#!python
q = QueryDict('foo=%00%7f%80%ff')
q['foo']
# actual: 
# u'\x00\x7f\x80\xff'
# expected:
# '\x00\x7f\x80\xff'
}}}

Relevant code:
https://github.com/django/django/blob/f89b11b879b83aa505dc8231da5f06ca4b1b062e/django/http/request.py#L397-L401

This was introduced by https://github.com/django/django/commit/fa02120d360387bebbbe735e86686bb4c7c43db2 while trying to accept broken user agents that send non-ascii as urlencoded raw values:
- the python 3 version seems fine: it tries to decode '''before''' calling the query string parser (from `urllib`)
- the python 2 version made a mistake: it decodes '''after''' calling the query string parser (and only on the value, not the key strangely)


I'm not sure how to fix this, as there are many locations where the key and value are re-encoded (`appendlist` does it too..)


In python 3 it fails too, but probably only because of the forced post-query-string-parser decoding into unicode string: to support binary values we don't want decoding at this point.

A test for python3:
{{{#!python
    def test_binary_input(self):
        """"""
        QueryDicts must be able to handle binary.
        """"""
        q = QueryDict(b'foo=%00%7f%80%ff')
        self.assertListEqual(list(map(ord, q['foo'])), [0x00, 0x7F, 0x80, 0xFF])
}}}
{{{
AssertionError: Lists differ: [0, 127, 65533, 65533] != [0, 127, 128, 255]

First differing element 2:
65533
128

- [0, 127, 65533, 65533]
+ [0, 127, 128, 255]
}}}"	Bug	closed	HTTP handling	1.11	Normal	wontfix		Claude Paroz	Unreviewed	0	0	0	0	0	0