FloatFields are converted to decimal and simplejson cannot serialize

Ugh, more decimal problems.

Hm - I can't seem to reproduce this. Can you verify that it's still happening and if so provide a minimal model and data that I can test against?

class Blah(models.Model):
        blub = models.FloatField(decimal_places=20, max_digits=27)

and the failing code:

def test():
        a = Blah()
        a.blub = 10980970.560287459837987986
        a.save()
        print serialize("json", Blah.objects.all())

note that this only happens on MySQL, it works fine in combination with SQLite

Hey Jacob,

I've been able to reproduce this in django trunk (currently r4463). Below is a simple test case to demonstrate the TypeError.

I think this arises when the database connector (postgresql_psycopg2 in my case) converts models.FloatField data to Python's decimal.Decimal type.

Demonstrate the error

Python 2.4.4c1 (#2, Oct 11 2006, 21:51:02) 
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from django.utils.simplejson import JSONEncoder
>>> from decimal import Decimal
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 312, in encode
    chunks = list(self.iterencode(o))
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 265, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 235, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 262, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 170, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 273, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 279, in _iterencode_default
    newobj = self.default(o)
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 300, in default
    raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("1.23") is not JSON serializable

Apply the patch

$ pwd
/home/ewalstad/var/django_trunk
$ patch django/utils/simplejson/encoder.py ./3324_Decimal_is_not_JSON_serializable.diff 
patching file django/utils/simplejson/encoder.py

Rerun the test

>>> from django.utils import simplejson
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
>>> print s
{"foo": [1.23]}
>>> d = simplejson.loads(s)
>>> print d
{u'foo': [1.23]}
>>> print type(d['foo'][0])
<type 'float'>

Note that upon deserialization the number that was a Decimal type becomes a float type. I found this comment by Federico Di Gregorio (from the psycopg team) saying that sending a float back into the db will work but there will be the expected/usual rounding errors.

Still, my vote is to commit this patch as it makes JSON work with PostgreSQL Numeric / Python Decimal data types where it doesn't work now.

I looked for where to add/change tests for this in Django source, but didn't find someplace that seemed simple and effective. If you need tests for this I'll write them but please give me a hint as to where the test should go (a subdir and/or file name would be great).

Thanks,

Eric.

Editable code to paste into the Python command line

from the make-it-dead-simple-for-the-django-dev dept:

from django.utils.simplejson import JSONEncoder from decimal import Decimal from django.utils import simplejson decimal_num = Decimal("1.23") s = JSONEncoder().encode({"foo": [decimal_num,]}) print s d = simplejson.loads(s) print d print type(d['foo'][0])

See also: Ticket #2365

I'm also seeing this in the current svn django, on a MySQL database, with a field specifying FloatField(max_digits=3, decimal_places=1,null=False):

>>> serializers.serialize('json',[my_orm_obj])
Traceback (most recent call last):
...
TypeError: Decimal("0.2") is not JSON serializable

Hello,

Since modifying the bundled simplejson library might not be very desirable, I would like to propose an alternative patch to the one submitted above by Eric. Django already subclasses "simplejson.JSONEncoder" in the file "django/core/serializers/json.py" to handle Date/Time types, so I believe it would be better to add a check there to also handle Decimal objects.

Since decimal is not included with Python 2.3 the patch does a simple check to see if it is available, and if it isn't, then it doesn't bother to check the object type. I also renamed this subclass because it no longer deals only with DateTime? objects.

I tested this patch on my machine and so far it works fine. I can now do a "./manage.py dumpdata --format=json app" when using a MySQL database.

I'll appreciate any comments on this patch.

Thanks!

Patches core/serializers/json.py instead of patching the bundled simplejson library.

Jorge Gajon's patch appears to be sweet'n'elegant, so I've marked that as ready for checkin, pending a core looking it over.

I was thinking that maybe it is not very efficient to check the presence of the Decimal library every time the encoding is called. It probably would be better to move this part to the top level of the module so that it is executed once (at module import).

decimal_available = False 
try: 
    import decimal 
    decimal_available = True 
except: 
    # decimal is not included in Python 2.3 
    pass 

I'm not uploading another attachment to avoid more clutter, and because I also would like to hear opinions :)

Thanks!

It would be nice to return a Decimal on deserialization, rather than returning a float as is done by the current patch.

(In [5302]) Fixed #2365, #3324 -- Renamed FloatField? to DecimalField? and changed the code to return Decimal instances in Python for this field. Backwards incompatible change.

Added a real FloatField? (stores floats in the database) and support for FloatField? and DecimalField? in newforms (analogous to IntegerField?).

Included decimal.py module (as django.utils._decimal) from Python 2.4. This is license compatible with Django and included for Python 2.3 compatibility only.

Large portions of this work are based on patches from Andy Durdin and Jorge Gajon.