Opened 9 years ago

Closed 8 years ago

#3324 closed (fixed)

FloatFields are converted to decimal and simplejson cannot serialize

Reported by: alex@… Owned by: jacob
Component: Core (Serialization) Version: master
Severity: Keywords: json decimal
Cc: mssnlayam@…, adurdin@…, eric@…, django@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

when serializing objects from a MySQL DB containing FloatFields to JSON, the float values are converted to Decimal objects, which are not understood by simplejson:

Traceback (most recent call last):
  File "blah/test.py", line 18, in ?
    test()
  File "blah/test.py", line 16, in test
    print serialize("json", Blah.objects.all())
  File "/home/alex/source/svn/django/django/core/serializers/__init__.py", line 55, in serialize
    s.serialize(queryset, **options)
  File "/home/alex/source/svn/django/django/core/serializers/base.py", line 49, in serialize
    self.end_serialization()
  File "/home/alex/source/svn/django/django/core/serializers/json.py", line 19, in end_serialization
    simplejson.dump(self.objects, self.stream, cls=DateTimeAwareJSONEncoder, **self.options)
  File "/home/alex/source/svn/django/django/utils/simplejson/__init__.py", line 119, in dump
    for chunk in iterable:
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 220, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 143, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 223, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 196, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 223, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 196, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 231, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 237, in _iterencode_default
    newobj = self.default(o)
  File "/home/alex/source/svn/django/django/core/serializers/json.py", line 51, in default
    return super(DateTimeAwareJSONEncoder, self).default(o)
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 258, in default
    raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("9999999.99999999999999999999") is not JSON serializable

Attachments (2)

3324_Decimal_is_not_JSON_serializable.diff (776 bytes) - added by eric@… 8 years ago.
serializers_json-Handle_Decimal.diff (1.7 KB) - added by Jorge Gajon <gajon@…> 8 years ago.
Patches core/serializers/json.py instead of patching the bundled simplejson library.

Download all attachments as: .zip

Change History (17)

comment:1 Changed 9 years ago by SmileyChris

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Summary changed from MySQL + FloatField + JSON => broken to FloatFields are converted to decimal and simplejson cannot serialize
  • Triage Stage changed from Unreviewed to Accepted

Ugh, more decimal problems.

comment:2 Changed 9 years ago by adurdin@…

  • Cc adurdin@… added

comment:3 Changed 9 years ago by jacob

  • Resolution set to worksforme
  • Status changed from new to closed

Hm - I can't seem to reproduce this. Can you verify that it's still happening and if so provide a minimal model and data that I can test against?

comment:4 Changed 9 years ago by alex@…

class Blah(models.Model):
        blub = models.FloatField(decimal_places=20, max_digits=27)

and the failing code:

def test():
        a = Blah()
        a.blub = 10980970.560287459837987986
        a.save()
        print serialize("json", Blah.objects.all())

note that this only happens on MySQL, it works fine in combination with SQLite

Changed 8 years ago by eric@…

comment:5 Changed 8 years ago by anonymous

  • Has patch set
  • Resolution worksforme deleted
  • Status changed from closed to reopened

Hey Jacob,

I've been able to reproduce this in django trunk (currently r4463). Below is a simple test case to demonstrate the TypeError.

I think this arises when the database connector (postgresql_psycopg2 in my case) converts models.FloatField data to Python's decimal.Decimal type.

Demonstrate the error

Python 2.4.4c1 (#2, Oct 11 2006, 21:51:02) 
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from django.utils.simplejson import JSONEncoder
>>> from decimal import Decimal
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 312, in encode
    chunks = list(self.iterencode(o))
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 265, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 235, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 262, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 170, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 273, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 279, in _iterencode_default
    newobj = self.default(o)
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 300, in default
    raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("1.23") is not JSON serializable

Apply the patch

$ pwd
/home/ewalstad/var/django_trunk
$ patch django/utils/simplejson/encoder.py ./3324_Decimal_is_not_JSON_serializable.diff 
patching file django/utils/simplejson/encoder.py

Rerun the test

>>> from django.utils import simplejson
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
>>> print s
{"foo": [1.23]}
>>> d = simplejson.loads(s)
>>> print d
{u'foo': [1.23]}
>>> print type(d['foo'][0])
<type 'float'>

Note that upon deserialization the number that was a Decimal type becomes a float type. I found this comment by Federico Di Gregorio (from the psycopg team) saying that sending a float back into the db will work but there will be the expected/usual rounding errors.

Still, my vote is to commit this patch as it makes JSON work with PostgreSQL Numeric / Python Decimal data types where it doesn't work now.

I looked for where to add/change tests for this in Django source, but didn't find someplace that seemed simple and effective. If you need tests for this I'll write them but please give me a hint as to where the test should go (a subdir and/or file name would be great).

Thanks,

Eric.

comment:6 Changed 8 years ago by anonymous

  • Cc eric@… added

comment:7 Changed 8 years ago by eric@…

Editable code to paste into the Python command line

from the make-it-dead-simple-for-the-django-dev dept:

comment:8 Changed 8 years ago by eric@…

See also: Ticket #2365

comment:9 Changed 8 years ago by yary h <not.com@…>

I'm also seeing this in the current svn django, on a MySQL database, with a field specifying FloatField(max_digits=3, decimal_places=1,null=False):

>>> serializers.serialize('json',[my_orm_obj])
Traceback (most recent call last):
...
TypeError: Decimal("0.2") is not JSON serializable

comment:10 Changed 8 years ago by Jorge Gajon <gajon@…>

  • Keywords json decimal added

Hello,

Since modifying the bundled simplejson library might not be very desirable, I would like to propose an alternative patch to the one submitted above by Eric. Django already subclasses "simplejson.JSONEncoder" in the file "django/core/serializers/json.py" to handle Date/Time types, so I believe it would be better to add a check there to also handle Decimal objects.

Since decimal is not included with Python 2.3 the patch does a simple check to see if it is available, and if it isn't, then it doesn't bother to check the object type. I also renamed this subclass because it no longer deals only with DateTime objects.

I tested this patch on my machine and so far it works fine. I can now do a "./manage.py dumpdata --format=json app" when using a MySQL database.

I'll appreciate any comments on this patch.

Thanks!

Changed 8 years ago by Jorge Gajon <gajon@…>

Patches core/serializers/json.py instead of patching the bundled simplejson library.

comment:11 Changed 8 years ago by Simon G. <dev@…>

  • Triage Stage changed from Accepted to Ready for checkin

Jorge Gajon's patch appears to be sweet'n'elegant, so I've marked that as ready for checkin, pending a core looking it over.

comment:12 Changed 8 years ago by Jorge Gajon <gajon@…>

I was thinking that maybe it is not very efficient to check the presence of the Decimal library every time the encoding is called. It probably would be better to move this part to the top level of the module so that it is executed once (at module import).

decimal_available = False 
try: 
    import decimal 
    decimal_available = True 
except: 
    # decimal is not included in Python 2.3 
    pass 

I'm not uploading another attachment to avoid more clutter, and because I also would like to hear opinions :)

Thanks!

comment:13 Changed 8 years ago by James Wheare <django@…>

  • Cc django@… added

comment:14 Changed 8 years ago by mssnlayam@…

  • Cc mssnlayam@… added

It would be nice to return a Decimal on deserialization, rather than returning a float as is done by the current patch.

comment:15 Changed 8 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [5302]) Fixed #2365, #3324 -- Renamed FloatField to DecimalField and changed the code
to return Decimal instances in Python for this field. Backwards incompatible
change.

Added a real FloatField (stores floats in the database) and support for
FloatField and DecimalField in newforms (analogous to IntegerField).

Included decimal.py module (as django.utils._decimal) from Python 2.4. This is
license compatible with Django and included for Python 2.3 compatibility only.

Large portions of this work are based on patches from Andy Durdin and Jorge
Gajon.

Note: See TracTickets for help on using tickets.
Back to Top