Code

Opened 18 months ago

Last modified 17 months ago

#19159 new Bug

loaddata reports DeserializationError instead of MemoryError

Reported by: django@… Owned by: nobody
Component: Core (Serialization) Version: 1.4
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The command manage.py loaddata data.json errors with DeserializationError instead of MemoryError.

files:

../lib/python2.7/site-packages/django/core/serializers/json.py
/usr/lib/python2.7/json/__init__.py

relevant code is simplejson.load(stream) inside the try:

def Deserializer(stream_or_string, **options):
    """
    Deserialize a stream or string of JSON data.
    """
    if isinstance(stream_or_string, basestring):
        stream = StringIO(stream_or_string)
    else:
        stream = stream_or_string
    try:
        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj
    except GeneratorExit:
        raise
    except Exception, e:
        # Map to deserializer error
        raise DeserializationError(e)

Attachments (0)

Change History (6)

comment:1 Changed 18 months ago by anonymous

  • Component changed from Uncategorized to Core (Serialization)
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 18 months ago by lrekucki

In Python 3 this could be written as raise DeserializationError from e, but I don't see a better equivalent in Python 2.

comment:3 Changed 18 months ago by anonymous

i think there's too much going on in the try statement:

        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj

the error arises from simplejson.load(stream) when reading a large json file from disk during fp.read().

comment:4 Changed 18 months ago by aaugustin

  • Triage Stage changed from Unreviewed to Accepted

comment:5 Changed 17 months ago by anonymous

Possibly fixed by #5423.

comment:6 Changed 17 months ago by claudep

No, #5423 is fixing dumpdata, this ticket is about loaddata. The problem with loaddata is that it would be very difficult to incrementally read the file.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as new
The owner will be changed from nobody to anonymous. Next status will be 'assigned'
as The resolution will be set. Next status will be 'closed'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.