(JSON)-serializing utf8 data fails

i haven't checked the django-fixture-code, but this problem is very similar to a problem with simplejson, so probably it is the cause:

with the simplejson serializer. like this example:

>>> from django.utils.simplejson import dumps,loads
>>>
>>> byte_text = '\xe7\x8c\xab' # the utf-8 representation of the japanese 'cat' character
>>> uni_text = byte_text.decode('utf-8')
>>> uni_text
u'\u732b'
>>>
>>> print loads(dumps(byte_text))
u'\xe7\x8c\xab'

and of course this is wrong. but:

>>> print loads(dumps(uni_text))
u'\u732b'

is ok.

so in short, when working with simplejson and non-ascii characters, then all strings that go into dumps have to be unicode-strings (not bytestrings)

I fixed this with the following simple patch:

--- Django-0.96/django/utils/simplejson/encoder.py      2007-01-31 00:34:15.000000000 +0200
+++ /usr/lib/python2.4/site-packages/django/utils/simplejson/encoder.py 2007-04-09 18:04:29.000000000 +0300
@@ -247,7 +247,7 @@ class JSONEncoder(object):
                 encoder = encode_basestring_ascii
             else:
                 encoder = encode_basestring
-            yield encoder(o)
+            yield encoder(o.decode('utf-8'))
         elif o is None:
             yield 'null'
         elif o is True:

i haven't tested the patch, but unfortunately there's a problem with it:

you're assuming that the bytestring-data the user has is encoded in UTF-8.

and that's not always true.

(another approach would be to use settings.DEFAULT_CHARSET, but that one is still not 100% correct)

but, to "bring" also good news, a django-branch has been created to switch it completely to unicode. with that done, this problem wouldn't be there.

This will be easiest to fix in the unicode branch. It's on the TODO list there. It's intended to be a short-lived sprinting branch, so I think it's best to leave this to be fixed there and then merged back.

The good news is that on that branch, your fix is absolutely the right idea, although we have some helper functions to make it easier.

Leaving the ticket open so that we remember to ensure it really is fixed.

uft8 problem with xml serializer

Saik: use the patch given above. Report back if you still have problems.

(In [5248]) unicode: Made the serializers unicode-aware. Refs #3878, #4227.

This was fixed in the unicode branch in [5248] (without changing simplejson.py at all, since that already works well with bytestrings and unicode). I'll close this ticket when the branch is merged back into trunk.

(In [5609]) Merged Unicode branch into trunk (r4952:5608). This should be fully backwards compatible for all practical purposes.

Fixed #2391, #2489, #2996, #3322, #3344, #3370, #3406, #3432, #3454, #3492, #3582, #3690, #3878, #3891, #3937, #4039, #4141, #4227, #4286, #4291, #4300, #4452, #4702