Opened 2 years ago

Last modified 4 months ago

#19159 new Bug

loaddata reports DeserializationError instead of MemoryError

Reported by: django@… Owned by: nobody
Component: Core (Serialization) Version: 1.4
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The command manage.py loaddata data.json errors with DeserializationError instead of MemoryError.

files:

../lib/python2.7/site-packages/django/core/serializers/json.py
/usr/lib/python2.7/json/__init__.py

relevant code is simplejson.load(stream) inside the try:

def Deserializer(stream_or_string, **options):
    """
    Deserialize a stream or string of JSON data.
    """
    if isinstance(stream_or_string, basestring):
        stream = StringIO(stream_or_string)
    else:
        stream = stream_or_string
    try:
        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj
    except GeneratorExit:
        raise
    except Exception, e:
        # Map to deserializer error
        raise DeserializationError(e)

Change History (7)

comment:1 Changed 2 years ago by anonymous

  • Component changed from Uncategorized to Core (Serialization)
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 2 years ago by lrekucki

In Python 3 this could be written as raise DeserializationError from e, but I don't see a better equivalent in Python 2.

comment:3 Changed 2 years ago by anonymous

i think there's too much going on in the try statement:

        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj

the error arises from simplejson.load(stream) when reading a large json file from disk during fp.read().

comment:4 Changed 2 years ago by aaugustin

  • Triage Stage changed from Unreviewed to Accepted

comment:5 Changed 2 years ago by anonymous

Possibly fixed by #5423.

comment:6 Changed 2 years ago by claudep

No, #5423 is fixing dumpdata, this ticket is about loaddata. The problem with loaddata is that it would be very difficult to incrementally read the file.

comment:7 Changed 4 months ago by berkerpeksag

json.Deserializer now uses six.reraise (see #18003). I think there are two questions that needs to be answered here:

  1. Should DeserializationError wrap MemoryError? If not, we need a patch to catch MemoryError explicitly.
  2. If yes, the ticket's title and description should be updated.

See also #12007.

Note: See TracTickets for help on using tickets.
Back to Top