[patch] is not JSON serializable

[Michael Samoylov]

Hi. I'm fighting with AJAX-based forms and dev process is stalled
because of such error:

Exception Type: TypeError
Exception Value:

'\xd0\x9e\xd0\xb1\xd1\x8f\xd0\xb7\xd0\xb0\xd1\x82\xd0\xb5\xd0\xbb\xd1\x8c\xd0\xbd\xd0\xbe\xd0\xb5

\xd0\xbf\xd0\xbe\xd0\xbb\xd0\xb5.' is not JSON serializable
Exception Location:

/usr/lib/python2.4/site-packages/django/utils/simplejson/encoder.py in

default, line 258

My code:

def registerAjax(request):

"""Allows user to register with CAPTCHA via AJAX call"""
if request.POST:

manipulator = EmailManipulator(request)
redirect_to = request.REQUEST.get(REDIRECT_FIELD_NAME, )
errors = manipulator.get_validation_errors(request.POST)
if not errors:

new_data = request.POST.copy()
try:

manipulator.save(new_data)
return HttpResponse(simplejson.dumps(('success',

redirect_to)), 'text/javascript')

except SMTPException, e:

return HttpResponse(simplejson.dumps(e),

'text/javascript')

else:

return HttpResponse(simplejson.dumps(errors),

'text/javascript')

But this code works perfect (returns json list of error messages)!

@login_required
def editInfoAjax(request):

if request.POST:

u = request.user
manipulator = Account.ChangeManipulator(u.id)
new_data = request.POST.copy()
errors = manipulator.get_validation_errors(new_data)
if not errors:

manipulator.do_html2python(new_data)
manipulator.save(new_data)
return HttpResponse(simplejson.dumps({'status':

'success'}), 'text/javascript')

else:

return HttpResponse(simplejson.dumps(errors),

'text/javascript')

I use django dev version from svn repository. I have LANGUAGE_CODE =
'ru' (have tryed to switch it to 'en' of course and got:

Exception Type: TypeError
Exception Value: 'This field is required.' is not JSON serializable

Any ideas? Thanks for your possible help

return HttpResponse(simplejson.dumps("%s" % errors), 'text/javascript') works but not as expected

[Michael Samoylov]

sorry for broken formatting, it was crosspoted from ​http://groups.google.com/group/django-users/browse_thread/thread/2c03c2d749c3c8d8

[Michael Samoylov]

Use gettext instead of gettext_lazy. Thanks, Ivan Sagalaev (Maniac@…)

[Ivan Sagalaev <Maniac@…>]

BTW, since it was the only place where gettext_lazy was used you could remove it from imports also...

[Michael Samoylov]

def encodeErrors(data):

"""
Prepares validator.errors for next usage with JSON.
Returns utf-8 encoded dictionary
"""
new_data = {}
for key, value in data.items():

new_data[key] = unicode(value[0], 'utf-8')

return new_data

[Michael Samoylov]

damn, broken formatting again

def encodeErrors(data):
    """
    Prepares validator.errors for next usage with JSON.
    Returns utf-8 encoded dictionary
    """
    new_data = {}
    for key, value in data.items():
        new_data[key] = unicode(value[0], 'utf-8')
    return new_data

[Ivan Sagalaev <Maniac@…>]

About this encodeErrors... It converts only the first item of error list which is wrong. Why not convert them all?

def encodeErrors(error_dict):
    """
    Converts error dict data into unciode
    """
    new_data = {}
    for key, value in error_dict.items():
        new_data[key] = [error.decode(settings.DEFAULT_CHARSET) for error in value]
    return new_data

Also your docstring says that it converts data into utf-8 while actually it converts them from utf-8 :-)

But in fact I think that instead of this function we should patch django's local copy of simplejson to convert all serialized strings into unicode using settings.DEFAULT_CHARSET. Since without it serialization just plain does not work for non-ascii letters.

[deryck@…]

I spent some time looking this one over, and I think this should be closed in favor of #2489. Once we have unicode, this problem goes away.

[deryck@…]

Changing the component and type and adding [patch]. Just cleaning up the report a bit to better indicate what is here.

[Ivan Sagalaev <Maniac@…>]

Deryck, I think your designation of this ticket for "Core framework" is wrong. The main issue is that JSON can't serialize error dict because of usage of gettext_lazy instead of gettext in one place. Unicode is another businness and the converting function in the comments is not even in the patch.

So this ticket is really just a plain fix for a small bug in validators.

[deryck@…]

I'll admit, it was hard for me to understand some of what the issue was here. :-) However, it looked to me like gettext_lazy was borking for the same reason, unicode characters. Perhaps, using the patch is harmless, though.

That's why I marked it "Core framework." The issue seemed to relate to serialization generally and unicode specifically, beyond just the simple patch. But I'm by no means the one to say. Was just trying to help clarify what seemed to be the problem.

[Ivan Sagalaev <Maniac@…>]

However, it looked to me like gettext_lazy was borking for the same reason, unicode characters

Seems like it's not... Generally simplejson works with byte strings but just outputs garbage (and may be this is locale related so may be if locale is equal to DEFAULT_CHARSET all will be fine). The issue with gettext_lazy (as I understand) is that simplejson sees this as a function, not a string, and refuses to serialize it. Replacing it with gettext makes actual translation of error message immediately so by the time when serialization is being made there is a plain string already.

AFAIK, a lazy version of gettext is only necessary when translating strings in models since by the time of import of a model you don't have a web environment yet and thus cannot know a user's preferred language. But validation works in view code so there's no such problem.

[limodou@…]

I'v also encounter this problem, so I write a simple function to convert string to unicode according settings.DEFAULT_CHARSET, the code is:

def uni_str(a, encoding):

if isinstance(a, (list, tuple)):

s = []
for i, k in enumerate(a):

s.append(uni_str(k, encoding))

return s

elif isinstance(a, dict):

s = {}
for i, k in enumerate(a.items()):

key, value = k
s[uni_str(key, encoding)] = uni_str(value, encoding)

return s

elif isinstance(a, unicode):

return a

elif isinstance(a, str) or (hasattr(a, 'str') and callable(getattr(a, 'str'))):

if getattr(a, 'str'):

a = str(a)

return unicode(a, encoding)

else:

return a

you should call it before you pass the data to simplejson. You can see that I test the obj's str method and use it to get a string.

[limodou@…]

I'm sorry for the code is messed:

def uni_str(a, encoding):
    if isinstance(a, (list, tuple)):
        s = []
        for i, k in enumerate(a):
            s.append(uni_str(k, encoding))
        return s
    elif isinstance(a, dict):
        s = {}
        for i, k in enumerate(a.items()):
            key, value = k
            s[uni_str(key, encoding)] = uni_str(value, encoding)
        return s
    elif isinstance(a, unicode):
        return a
    elif isinstance(a, str) or (hasattr(a, '__str__') and callable(getattr(a, '__str__'))):
        if getattr(a, '__str__'):
            a = str(a)
        return unicode(a, encoding)
    else:
        return a

[Malcolm Tredinnick]

(In [3816]) Fixed #2727 -- Fixed problem with serialising error messages from validation.
Thanks, Ivan Saglaev.

[Ivan Sagalaev <Maniac@…>]

To give proper credit my only help was only vague idea of why this could not work, the actual patch is by msamoylov :-)