[patch] Django unicode patches

[Victor Ng <victor.ng@…>]

This patch supercedes the patches in #2543, #2554.

I've removed the body of encode_output from VariableNode as it's not correct. We should be maintaining unicode strings throughout the Django code and converting to a charset only at the last possible moment.

I've modified django.http.init to encode the content to settings.DEFAULT_CHARSET which so that we never get a case of mixed unicode/non-unicode strings.

[Victor Ng <victor.ng@…>]

patch against django r3641 to fix various unicode problems

[Victor Ng <victor.ng@…>]

additional fix for FormWrapper objects in the context. patched against django r3645

[Victor Ng <victor.ng@…>]

I've got a bunch of unit tests in my app that excercise this stuff to make sure it works, still have to pull them out into a patch against django though.

[hugo]

I move this back to adrian, as it patches many core files which aren't really part of the i18n subproject.

[anonymous]

I tried applying to 0.95 but it failed, even when fixing obvious stuff manually ... i get unicodedecodeerrors (even on completely cleanly created database using syncdb)

Can we expect any progress on this one soon?

[Victor Ng <victor.ng@…>]

After wrestling with the Unicode errors in Django for about 6 months now, I've just settled on using a middleware layer to decode the input string using the settings.DEFAULT_CHARSET and then re-encoding the data back to UTF-8.

If you modify the request.POST and the request.GET MultiValueDict instances as well as the request.path, you'll get ASCII clean UTF-8 data guaranteed to flow through Django.

This 'solves' the unicode problem in the existing 0.95 codebase by forcing all internal data to be UTF-8 data. This way you don't have to modify calls of str() to unicode(), you don't have to use the "u" qualifier for string data and you don't have to worry about what any particular database backend is going to do.

In other words, you don't have to touch any other parts of django and as long as you assume that internal data is UTF8 and not just plain ASCII, everything should work fine.

[Thomas Güttler <hv@…>]

There is now a unicode branch:

​http://code.djangoproject.com/wiki/UnicodeBranch