FloatFields are converted to decimal and simplejson cannot serialize

[alex@…]

when serializing objects from a MySQL DB containing FloatFields to JSON, the float values are converted to Decimal objects, which are not understood by simplejson:

Traceback (most recent call last):
  File "blah/test.py", line 18, in ?
    test()
  File "blah/test.py", line 16, in test
    print serialize("json", Blah.objects.all())
  File "/home/alex/source/svn/django/django/core/serializers/__init__.py", line 55, in serialize
    s.serialize(queryset, **options)
  File "/home/alex/source/svn/django/django/core/serializers/base.py", line 49, in serialize
    self.end_serialization()
  File "/home/alex/source/svn/django/django/core/serializers/json.py", line 19, in end_serialization
    simplejson.dump(self.objects, self.stream, cls=DateTimeAwareJSONEncoder, **self.options)
  File "/home/alex/source/svn/django/django/utils/simplejson/__init__.py", line 119, in dump
    for chunk in iterable:
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 220, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 143, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 223, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 196, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 223, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 196, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 231, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 237, in _iterencode_default
    newobj = self.default(o)
  File "/home/alex/source/svn/django/django/core/serializers/json.py", line 51, in default
    return super(DateTimeAwareJSONEncoder, self).default(o)
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 258, in default
    raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("9999999.99999999999999999999") is not JSON serializable

[Chris Beaven]

Ugh, more decimal problems.

[Jacob]

Hm - I can't seem to reproduce this. Can you verify that it's still happening and if so provide a minimal model and data that I can test against?

[alex@…]

class Blah(models.Model):
        blub = models.FloatField(decimal_places=20, max_digits=27)

and the failing code:

def test():
        a = Blah()
        a.blub = 10980970.560287459837987986
        a.save()
        print serialize("json", Blah.objects.all())

note that this only happens on MySQL, it works fine in combination with SQLite

[anonymous]

Hey Jacob,

I've been able to reproduce this in django trunk (currently r4463). Below is a simple test case to demonstrate the TypeError.

I think this arises when the database connector (postgresql_psycopg2 in my case) converts models.FloatField data to Python's decimal.Decimal type.

Demonstrate the error

Python 2.4.4c1 (#2, Oct 11 2006, 21:51:02) 
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from django.utils.simplejson import JSONEncoder
>>> from decimal import Decimal
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 312, in encode
    chunks = list(self.iterencode(o))
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 265, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 235, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 262, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 170, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 273, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 279, in _iterencode_default
    newobj = self.default(o)
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 300, in default
    raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("1.23") is not JSON serializable

Apply the patch

$ pwd
/home/ewalstad/var/django_trunk
$ patch django/utils/simplejson/encoder.py ./3324_Decimal_is_not_JSON_serializable.diff 
patching file django/utils/simplejson/encoder.py

Rerun the test

>>> from django.utils import simplejson
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
>>> print s
{"foo": [1.23]}
>>> d = simplejson.loads(s)
>>> print d
{u'foo': [1.23]}
>>> print type(d['foo'][0])
<type 'float'>

Note that upon deserialization the number that was a Decimal type becomes a float type. I found ​this comment by Federico Di Gregorio (from the psycopg team) saying that sending a float back into the db will work but there will be the expected/usual rounding errors.

Still, my vote is to commit this patch as it makes JSON work with PostgreSQL Numeric / Python Decimal data types where it doesn't work now.

I looked for where to add/change tests for this in Django source, but didn't find someplace that seemed simple and effective. If you need tests for this I'll write them but please give me a hint as to where the test should go (a subdir and/or file name would be great).

Thanks,

Eric.

[eric@…]

Editable code to paste into the Python command line

from the make-it-dead-simple-for-the-django-dev dept:

from django.utils.simplejson import JSONEncoder from decimal import Decimal from django.utils import simplejson decimal_num = Decimal("1.23") s = JSONEncoder().encode({"foo": [decimal_num,]}) print s d = simplejson.loads(s) print d print type(d['foo'][0])

[eric@…]

See also: Ticket #2365

[yary h <not.com@…>]

I'm also seeing this in the current svn django, on a MySQL database, with a field specifying FloatField(max_digits=3, decimal_places=1,null=False):

>>> serializers.serialize('json',[my_orm_obj])
Traceback (most recent call last):
...
TypeError: Decimal("0.2") is not JSON serializable

[Jorge Gajon <gajon@…>]

Hello,

Since modifying the bundled simplejson library might not be very desirable, I would like to propose an alternative patch to the one submitted above by Eric. Django already subclasses "simplejson.JSONEncoder" in the file "django/core/serializers/json.py" to handle Date/Time types, so I believe it would be better to add a check there to also handle Decimal objects.

Since decimal is not included with Python 2.3 the patch does a simple check to see if it is available, and if it isn't, then it doesn't bother to check the object type. I also renamed this subclass because it no longer deals only with DateTime objects.

I tested this patch on my machine and so far it works fine. I can now do a "./manage.py dumpdata --format=json app" when using a MySQL database.

I'll appreciate any comments on this patch.

Thanks!

[Jorge Gajon <gajon@…>]

Patches core/serializers/json.py instead of patching the bundled simplejson library.

[Simon G. <dev@…>]

Jorge Gajon's patch appears to be sweet'n'elegant, so I've marked that as ready for checkin, pending a core looking it over.

[Jorge Gajon <gajon@…>]

I was thinking that maybe it is not very efficient to check the presence of the Decimal library every time the encoding is called. It probably would be better to move this part to the top level of the module so that it is executed once (at module import).

decimal_available = False 
try: 
    import decimal 
    decimal_available = True 
except: 
    # decimal is not included in Python 2.3 
    pass 

I'm not uploading another attachment to avoid more clutter, and because I also would like to hear opinions :)

Thanks!

[mssnlayam@…]

It would be nice to return a Decimal on deserialization, rather than returning a float as is done by the current patch.

[Malcolm Tredinnick]

(In [5302]) Fixed #2365, #3324 -- Renamed FloatField to DecimalField and changed the code
to return Decimal instances in Python for this field. Backwards incompatible
change.

Added a real FloatField (stores floats in the database) and support for
FloatField and DecimalField in newforms (analogous to IntegerField).

Included decimal.py module (as django.utils._decimal) from Python 2.4. This is
license compatible with Django and included for Python 2.3 compatibility only.

Large portions of this work are based on patches from Andy Durdin and Jorge
Gajon.