Opened 17 years ago

Closed 17 years ago

#3324 closed (fixed)

FloatFields are converted to decimal and simplejson cannot serialize

Reported by: alex@… Owned by: Jacob
Component: Core (Serialization) Version: dev
Severity: Keywords: json decimal
Cc: mssnlayam@…, adurdin@…, eric@…, django@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

when serializing objects from a MySQL DB containing FloatFields to JSON, the float values are converted to Decimal objects, which are not understood by simplejson:

Traceback (most recent call last):
  File "blah/test.py", line 18, in ?
    test()
  File "blah/test.py", line 16, in test
    print serialize("json", Blah.objects.all())
  File "/home/alex/source/svn/django/django/core/serializers/__init__.py", line 55, in serialize
    s.serialize(queryset, **options)
  File "/home/alex/source/svn/django/django/core/serializers/base.py", line 49, in serialize
    self.end_serialization()
  File "/home/alex/source/svn/django/django/core/serializers/json.py", line 19, in end_serialization
    simplejson.dump(self.objects, self.stream, cls=DateTimeAwareJSONEncoder, **self.options)
  File "/home/alex/source/svn/django/django/utils/simplejson/__init__.py", line 119, in dump
    for chunk in iterable:
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 220, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 143, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 223, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 196, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 223, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 196, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 231, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 237, in _iterencode_default
    newobj = self.default(o)
  File "/home/alex/source/svn/django/django/core/serializers/json.py", line 51, in default
    return super(DateTimeAwareJSONEncoder, self).default(o)
  File "/home/alex/source/svn/django/django/utils/simplejson/encoder.py", line 258, in default
    raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("9999999.99999999999999999999") is not JSON serializable

Attachments (2)

3324_Decimal_is_not_JSON_serializable.diff (776 bytes ) - added by eric@… 17 years ago.
serializers_json-Handle_Decimal.diff (1.7 KB ) - added by Jorge Gajon <gajon@…> 17 years ago.
Patches core/serializers/json.py instead of patching the bundled simplejson library.

Download all attachments as: .zip

Change History (17)

comment:1 by Chris Beaven, 17 years ago

Summary: MySQL + FloatField + JSON => brokenFloatFields are converted to decimal and simplejson cannot serialize
Triage Stage: UnreviewedAccepted

Ugh, more decimal problems.

comment:2 by adurdin@…, 17 years ago

Cc: adurdin@… added

comment:3 by Jacob, 17 years ago

Resolution: worksforme
Status: newclosed

Hm - I can't seem to reproduce this. Can you verify that it's still happening and if so provide a minimal model and data that I can test against?

comment:4 by alex@…, 17 years ago

class Blah(models.Model):
        blub = models.FloatField(decimal_places=20, max_digits=27)

and the failing code:

def test():
        a = Blah()
        a.blub = 10980970.560287459837987986
        a.save()
        print serialize("json", Blah.objects.all())

note that this only happens on MySQL, it works fine in combination with SQLite

comment:5 by anonymous, 17 years ago

Has patch: set
Resolution: worksforme
Status: closedreopened

Hey Jacob,

I've been able to reproduce this in django trunk (currently r4463). Below is a simple test case to demonstrate the TypeError.

I think this arises when the database connector (postgresql_psycopg2 in my case) converts models.FloatField data to Python's decimal.Decimal type.

Demonstrate the error

Python 2.4.4c1 (#2, Oct 11 2006, 21:51:02) 
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from django.utils.simplejson import JSONEncoder
>>> from decimal import Decimal
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 312, in encode
    chunks = list(self.iterencode(o))
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 265, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 235, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 262, in _iterencode
    for chunk in self._iterencode_list(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 170, in _iterencode_list
    for chunk in self._iterencode(value, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 273, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 279, in _iterencode_default
    newobj = self.default(o)
  File "/home/ewalstad/var/django_trunk/django/utils/simplejson/encoder.py", line 300, in default
    raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("1.23") is not JSON serializable

Apply the patch

$ pwd
/home/ewalstad/var/django_trunk
$ patch django/utils/simplejson/encoder.py ./3324_Decimal_is_not_JSON_serializable.diff 
patching file django/utils/simplejson/encoder.py

Rerun the test

>>> from django.utils import simplejson
>>> decimal_num = Decimal("1.23")
>>> s = JSONEncoder().encode({"foo": [decimal_num,]})
>>> print s
{"foo": [1.23]}
>>> d = simplejson.loads(s)
>>> print d
{u'foo': [1.23]}
>>> print type(d['foo'][0])
<type 'float'>

Note that upon deserialization the number that was a Decimal type becomes a float type. I found this comment by Federico Di Gregorio (from the psycopg team) saying that sending a float back into the db will work but there will be the expected/usual rounding errors.

Still, my vote is to commit this patch as it makes JSON work with PostgreSQL Numeric / Python Decimal data types where it doesn't work now.

I looked for where to add/change tests for this in Django source, but didn't find someplace that seemed simple and effective. If you need tests for this I'll write them but please give me a hint as to where the test should go (a subdir and/or file name would be great).

Thanks,

Eric.

comment:6 by anonymous, 17 years ago

Cc: eric@… added

comment:7 by eric@…, 17 years ago

Editable code to paste into the Python command line

from the make-it-dead-simple-for-the-django-dev dept:

comment:8 by eric@…, 17 years ago

See also: Ticket #2365

comment:9 by yary h <not.com@…>, 17 years ago

I'm also seeing this in the current svn django, on a MySQL database, with a field specifying FloatField(max_digits=3, decimal_places=1,null=False):

>>> serializers.serialize('json',[my_orm_obj])
Traceback (most recent call last):
...
TypeError: Decimal("0.2") is not JSON serializable

comment:10 by Jorge Gajon <gajon@…>, 17 years ago

Keywords: json decimal added

Hello,

Since modifying the bundled simplejson library might not be very desirable, I would like to propose an alternative patch to the one submitted above by Eric. Django already subclasses "simplejson.JSONEncoder" in the file "django/core/serializers/json.py" to handle Date/Time types, so I believe it would be better to add a check there to also handle Decimal objects.

Since decimal is not included with Python 2.3 the patch does a simple check to see if it is available, and if it isn't, then it doesn't bother to check the object type. I also renamed this subclass because it no longer deals only with DateTime objects.

I tested this patch on my machine and so far it works fine. I can now do a "./manage.py dumpdata --format=json app" when using a MySQL database.

I'll appreciate any comments on this patch.

Thanks!

by Jorge Gajon <gajon@…>, 17 years ago

Patches core/serializers/json.py instead of patching the bundled simplejson library.

comment:11 by Simon G. <dev@…>, 17 years ago

Triage Stage: AcceptedReady for checkin

Jorge Gajon's patch appears to be sweet'n'elegant, so I've marked that as ready for checkin, pending a core looking it over.

comment:12 by Jorge Gajon <gajon@…>, 17 years ago

I was thinking that maybe it is not very efficient to check the presence of the Decimal library every time the encoding is called. It probably would be better to move this part to the top level of the module so that it is executed once (at module import).

decimal_available = False 
try: 
    import decimal 
    decimal_available = True 
except: 
    # decimal is not included in Python 2.3 
    pass 

I'm not uploading another attachment to avoid more clutter, and because I also would like to hear opinions :)

Thanks!

comment:13 by James Wheare <django@…>, 17 years ago

Cc: django@… added

comment:14 by mssnlayam@…, 17 years ago

Cc: mssnlayam@… added

It would be nice to return a Decimal on deserialization, rather than returning a float as is done by the current patch.

comment:15 by Malcolm Tredinnick, 17 years ago

Resolution: fixed
Status: reopenedclosed

(In [5302]) Fixed #2365, #3324 -- Renamed FloatField to DecimalField and changed the code
to return Decimal instances in Python for this field. Backwards incompatible
change.

Added a real FloatField (stores floats in the database) and support for
FloatField and DecimalField in newforms (analogous to IntegerField).

Included decimal.py module (as django.utils._decimal) from Python 2.4. This is
license compatible with Django and included for Python 2.3 compatibility only.

Large portions of this work are based on patches from Andy Durdin and Jorge
Gajon.

Note: See TracTickets for help on using tickets.
Back to Top