Opened 5 years ago

Last modified 3 years ago

#19159 new Bug

loaddata reports DeserializationError instead of MemoryError

Reported by: django@… Owned by: nobody
Component: Core (Serialization) Version: 1.4
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


The command loaddata data.json errors with DeserializationError instead of MemoryError.



relevant code is simplejson.load(stream) inside the try:

def Deserializer(stream_or_string, **options):
    Deserialize a stream or string of JSON data.
    if isinstance(stream_or_string, basestring):
        stream = StringIO(stream_or_string)
        stream = stream_or_string
        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj
    except GeneratorExit:
    except Exception, e:
        # Map to deserializer error
        raise DeserializationError(e)

Change History (7)

comment:1 Changed 5 years ago by anonymous

Component: UncategorizedCore (Serialization)

comment:2 Changed 5 years ago by Łukasz Rekucki

In Python 3 this could be written as raise DeserializationError from e, but I don't see a better equivalent in Python 2.

comment:3 Changed 5 years ago by anonymous

i think there's too much going on in the try statement:

        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj

the error arises from simplejson.load(stream) when reading a large json file from disk during

comment:4 Changed 5 years ago by Aymeric Augustin

Triage Stage: UnreviewedAccepted

comment:5 Changed 5 years ago by anonymous

Possibly fixed by #5423.

comment:6 Changed 5 years ago by Claude Paroz

No, #5423 is fixing dumpdata, this ticket is about loaddata. The problem with loaddata is that it would be very difficult to incrementally read the file.

comment:7 Changed 3 years ago by Berker Peksag

json.Deserializer now uses six.reraise (see #18003). I think there are two questions that needs to be answered here:

  1. Should DeserializationError wrap MemoryError? If not, we need a patch to catch MemoryError explicitly.
  2. If yes, the ticket's title and description should be updated.

See also #12007.

Note: See TracTickets for help on using tickets.
Back to Top