Opened 9 years ago

Closed 9 years ago

Last modified 6 years ago

#2727 closed defect (fixed)

[patch] is not JSON serializable

Reported by: msamoylov Owned by: adrian
Component: Validators Version: master
Severity: normal Keywords:
Cc: Maniac@… Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Hi. I'm fighting with AJAX-based forms and dev process is stalled
because of such error:

Exception Type: TypeError
Exception Value:

'\xd0\x9e\xd0\xb1\xd1\x8f\xd0\xb7\xd0\xb0\xd1\x82\xd0\xb5\xd0\xbb\xd1\x8c\xd0\xbd\xd0\xbe\xd0\xb5

\xd0\xbf\xd0\xbe\xd0\xbb\xd0\xb5.' is not JSON serializable
Exception Location:

/usr/lib/python2.4/site-packages/django/utils/simplejson/encoder.py in

default, line 258

My code:

def registerAjax(request):

"""Allows user to register with CAPTCHA via AJAX call"""
if request.POST:

manipulator = EmailManipulator(request)
redirect_to = request.REQUEST.get(REDIRECT_FIELD_NAME, )
errors = manipulator.get_validation_errors(request.POST)
if not errors:

new_data = request.POST.copy()
try:

manipulator.save(new_data)
return HttpResponse(simplejson.dumps(('success',

redirect_to)), 'text/javascript')

except SMTPException, e:

return HttpResponse(simplejson.dumps(e),

'text/javascript')

else:

return HttpResponse(simplejson.dumps(errors),

'text/javascript')

But this code works perfect (returns json list of error messages)!

@login_required
def editInfoAjax(request):

if request.POST:

u = request.user
manipulator = Account.ChangeManipulator(u.id)
new_data = request.POST.copy()
errors = manipulator.get_validation_errors(new_data)
if not errors:

manipulator.do_html2python(new_data)
manipulator.save(new_data)
return HttpResponse(simplejson.dumps({'status':

'success'}), 'text/javascript')

else:

return HttpResponse(simplejson.dumps(errors),

'text/javascript')

I use django dev version from svn repository. I have LANGUAGE_CODE =
'ru' (have tryed to switch it to 'en' of course and got:

Exception Type: TypeError
Exception Value: 'This field is required.' is not JSON serializable

Any ideas? Thanks for your possible help

return HttpResponse(simplejson.dumps("%s" % errors), 'text/javascript') works but not as expected

Attachments (1)

gettext.diff (640 bytes) - added by msamoylov 9 years ago.
Use gettext instead of gettext_lazy. Thanks, Ivan Sagalaev (Maniac@…)

Download all attachments as: .zip

Change History (15)

comment:1 Changed 9 years ago by msamoylov

Changed 9 years ago by msamoylov

Use gettext instead of gettext_lazy. Thanks, Ivan Sagalaev (Maniac@…)

comment:2 Changed 9 years ago by Ivan Sagalaev <Maniac@…>

  • Cc Maniac@… added

BTW, since it was the only place where gettext_lazy was used you could remove it from imports also...

comment:3 Changed 9 years ago by msamoylov

def encodeErrors(data):

"""
Prepares validator.errors for next usage with JSON.
Returns utf-8 encoded dictionary
"""
new_data = {}
for key, value in data.items():

new_data[key] = unicode(value[0], 'utf-8')

return new_data

comment:4 Changed 9 years ago by msamoylov

damn, broken formatting again

def encodeErrors(data):
    """
    Prepares validator.errors for next usage with JSON.
    Returns utf-8 encoded dictionary
    """
    new_data = {}
    for key, value in data.items():
        new_data[key] = unicode(value[0], 'utf-8')
    return new_data

comment:5 Changed 9 years ago by Ivan Sagalaev <Maniac@…>

About this encodeErrors... It converts only the first item of error list which is wrong. Why not convert them all?

def encodeErrors(error_dict):
    """
    Converts error dict data into unciode
    """
    new_data = {}
    for key, value in error_dict.items():
        new_data[key] = [error.decode(settings.DEFAULT_CHARSET) for error in value]
    return new_data

Also your docstring says that it converts data into utf-8 while actually it converts them from utf-8 :-)

But in fact I think that instead of this function we should patch django's local copy of simplejson to convert all serialized strings into unicode using settings.DEFAULT_CHARSET. Since without it serialization just plain does not work for non-ascii letters.

comment:6 Changed 9 years ago by deryck@…

I spent some time looking this one over, and I think this should be closed in favor of #2489. Once we have unicode, this problem goes away.

comment:7 Changed 9 years ago by deryck@…

  • Component changed from Validators to Core framework
  • Summary changed from is not JSON serializable to [patch] is not JSON serializable
  • Type changed from enhancement to defect

Changing the component and type and adding [patch]. Just cleaning up the report a bit to better indicate what is here.

comment:8 Changed 9 years ago by Ivan Sagalaev <Maniac@…>

  • Component changed from Core framework to Validators

Deryck, I think your designation of this ticket for "Core framework" is wrong. The main issue is that JSON can't serialize error dict because of usage of gettext_lazy instead of gettext in one place. Unicode is another businness and the converting function in the comments is not even in the patch.

So this ticket is really just a plain fix for a small bug in validators.

comment:9 Changed 9 years ago by deryck@…

I'll admit, it was hard for me to understand some of what the issue was here. :-) However, it looked to me like gettext_lazy was borking for the same reason, unicode characters. Perhaps, using the patch is harmless, though.

That's why I marked it "Core framework." The issue seemed to relate to serialization generally and unicode specifically, beyond just the simple patch. But I'm by no means the one to say. Was just trying to help clarify what seemed to be the problem.

comment:10 Changed 9 years ago by Ivan Sagalaev <Maniac@…>

However, it looked to me like gettext_lazy was borking for the same reason, unicode characters

Seems like it's not... Generally simplejson works with byte strings but just outputs garbage (and may be this is locale related so may be if locale is equal to DEFAULT_CHARSET all will be fine). The issue with gettext_lazy (as I understand) is that simplejson sees this as a function, not a string, and refuses to serialize it. Replacing it with gettext makes actual translation of error message immediately so by the time when serialization is being made there is a plain string already.

AFAIK, a lazy version of gettext is only necessary when translating strings in models since by the time of import of a model you don't have a web environment yet and thus cannot know a user's preferred language. But validation works in view code so there's no such problem.

comment:11 Changed 9 years ago by limodou@…

I'v also encounter this problem, so I write a simple function to convert string to unicode according settings.DEFAULT_CHARSET, the code is:

def uni_str(a, encoding):

if isinstance(a, (list, tuple)):

s = []
for i, k in enumerate(a):

s.append(uni_str(k, encoding))

return s

elif isinstance(a, dict):

s = {}
for i, k in enumerate(a.items()):

key, value = k
s[uni_str(key, encoding)] = uni_str(value, encoding)

return s

elif isinstance(a, unicode):

return a

elif isinstance(a, str) or (hasattr(a, 'str') and callable(getattr(a, 'str'))):

if getattr(a, 'str'):

a = str(a)

return unicode(a, encoding)

else:

return a

you should call it before you pass the data to simplejson. You can see that I test the obj's str method and use it to get a string.

comment:12 Changed 9 years ago by limodou@…

I'm sorry for the code is messed:

def uni_str(a, encoding):
    if isinstance(a, (list, tuple)):
        s = []
        for i, k in enumerate(a):
            s.append(uni_str(k, encoding))
        return s
    elif isinstance(a, dict):
        s = {}
        for i, k in enumerate(a.items()):
            key, value = k
            s[uni_str(key, encoding)] = uni_str(value, encoding)
        return s
    elif isinstance(a, unicode):
        return a
    elif isinstance(a, str) or (hasattr(a, '__str__') and callable(getattr(a, '__str__'))):
        if getattr(a, '__str__'):
            a = str(a)
        return unicode(a, encoding)
    else:
        return a

comment:13 Changed 9 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [3816]) Fixed #2727 -- Fixed problem with serialising error messages from validation.
Thanks, Ivan Saglaev.

comment:14 Changed 9 years ago by Ivan Sagalaev <Maniac@…>

To give proper credit my only help was only vague idea of why this could not work, the actual patch is by msamoylov :-)

Note: See TracTickets for help on using tickets.
Back to Top