Test Client fails to decode json response with unicode characters

[Aniruddha Maru]

Since application/json response ought to be encoded with utf-8 charset per spec ​RFC 7159, they should be decoded as such.

The problematic line is ​here:

            response._json = json.loads(response.content.decode(), **extra)

Which ought to be changed to:

            response._json = json.loads(response.content.decode('utf-8'), **extra)

[Aniruddha Maru]

This is only relevant for Py2.

[Aniruddha Maru]

patch.diff

[Aniruddha Maru]

patch_stable/1.10.x.diff

[Claude Paroz]

I think Django makes the assumption that when Unicode characters are sent JSON-encoded on the wire, those characters are escaped (\\u00e9).
Is that a wrong assumption?

[Tim Graham]

Claude, could you provide an explanation of why Django would make that assumption? I'm not knowledgeable to confirm or reject it.

[Claude Paroz]

Tim, I'm not talking about master or Python 3, but with Python 2, calling decode() without specifying an encoding will default to 'ascii' decoding.

[Tim Graham]

Since response.json is new in 1.9, even if this is valid, it doesn't qualify for a backport at this time. Marking as "fixed" since Django's master branch only supports Python 3.

[Claude Paroz]

I do not totally agree. If this is recognized as a bug, as 1.11 still supports Python 2 and considering we are still in beta phase, this should be fixed for 1.11.
The proposed patch is simple and has a test. I'm available to create the PR.

[Tim Graham]

The guideline for the post-beta period is only to backport release blocking bugs (if this is considered a blocker, it would normally also be fixed in 1.10) but I'm fine with whatever you want to do here.

[Tim Graham]

Aniruddha, could you gave the use case of using ensure_ascii=False in the JsonResponse?

[Aniruddha Maru]

Tim, it's really straightforward: we have browser based API documentation, and unicode data. Since json supports utf-8 encoding, it's pretty standard to always return unicode, rather than have an encode-decode cycle.

Here's some more context of other people who've talked about this bug:
​https://github.com/tomchristie/django-rest-framework/issues/2891
​https://github.com/tomchristie/django-rest-framework/issues/2891#issuecomment-178932902

Django rest framework also offers a flag to disable this behavior: ​http://www.django-rest-framework.org/api-guide/settings/#unicode_json, which is enabled by default.

[Tim Graham]

Okay, thanks. Can you create the pull request against stable/1.11.x? I think you could modify the existing json view to return some non-ASCII content rather than adding another view.

[Tim Graham]

Claude, do you plan to address this for 1.11?

[Tim Graham]

Reclosing. A pull request was never provided for 1.11.

[Fabio Bonelli]

Pull request for 1.11: ​https://github.com/django/django/pull/9061

[Tim Graham]

As this isn't a regression or a bug in a new feature in 1.11, it doesn't qualify for a backport to stable/1.11.x. See our ​supported versions policy.

[Fabio Bonelli]

You gave somewhat of a mixed signal with comment:11.

Anyway, this bug report has still a patch attached for master. Would you accept a PR against that branch?

[Tim Graham]

Sorry for the confusion. 6 months ago when I made that comment, 1.11 wasn't released so less critical bug fixes were more acceptable.

Master doesn't support Python 2 so as far as I understand, this issue doesn't affect it.