id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
10571	FakePayload Truncates Unicode Content	rwagner@…	nobody	"Hi,

While testing an API for posting new objects using JSON, I think I've tripped over a problem with cStringIO and Unicode. My test had a few line like this, and the {{{raw_post_data}}} in the receiving view was truncated.

{{{
#!python
    json = u'{""name"": ""Rick""}'
    response = self.c.post('/api/person/', json,
                            content_type=""application/json"")
}}}

Looking a little closer, it seems that cStringIO doesn't treat Unicode objects very well. In particular, because the codec is utf-16, the number of bytes required for reading is doubled.

Here's a demonstration (I hope it's sufficient):

{{{
#!python
>>> from django.test.client import FakePayload
>>> json = u'{""name"": ""Rick""}'
>>> payload = FakePayload(json)
>>> len(json)
16
>>> payload._FakePayload__len
16
>>> payload.read()
'\x00{\x00""\x00n\x00a\x00m\x00e\x00""\x00:'
>>> '\x00{\x00""\x00n\x00a\x00m\x00e\x00""\x00:'.decode(""utf-16"")
u'{""name"":'
>>> import cStringIO
>>> import StringIO
>>> cStringIO.StringIO(json).read()
'\x00{\x00""\x00n\x00a\x00m\x00e\x00""\x00:\x00 \x00""\x00R\x00i\x00c\x00k\x00""\x00}'
>>> len(cStringIO.StringIO(json).read())
32
>>> StringIO.StringIO(json).read()
u'{""name"": ""Rick""}'
>>> len(StringIO.StringIO(json).read())
16
>>> cStringIO.StringIO(json).read(16)
'\x00{\x00""\x00n\x00a\x00m\x00e\x00""\x00:'
>>> cStringIO.StringIO(json).read(16).decode(""utf-16"")
u'{""name"":'
>>> 
}}}

A direct fix would be to use StringIO instead of cStringIO, but I appreciate the desire to keep the speed-up, especially for unit testing.

Please correct me if I've misinterpreted the use POST in the test client.

Thanks,
Rick

PS: This may be related to #9480."	Bug	closed	Testing framework	dev	Normal	fixed		Anand Kumria Peter van Kampen	Ready for checkin	1	0	0	0	0	0