loaddata reports DeserializationError instead of MemoryError

[django@…]

The command manage.py loaddata data.json errors with DeserializationError instead of MemoryError.

files:

../lib/python2.7/site-packages/django/core/serializers/json.py
/usr/lib/python2.7/json/__init__.py

relevant code is simplejson.load(stream) inside the try:

def Deserializer(stream_or_string, **options):
    """
    Deserialize a stream or string of JSON data.
    """
    if isinstance(stream_or_string, basestring):
        stream = StringIO(stream_or_string)
    else:
        stream = stream_or_string
    try:
        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj
    except GeneratorExit:
        raise
    except Exception, e:
        # Map to deserializer error
        raise DeserializationError(e)

[Łukasz Rekucki]

In Python 3 this could be written as raise DeserializationError from e, but I don't see a better equivalent in Python 2.

[anonymous]

i think there's too much going on in the try statement:

        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj

the error arises from simplejson.load(stream) when reading a large json file from disk during fp.read().

[anonymous]

Possibly fixed by #5423.

[Claude Paroz]

No, #5423 is fixing dumpdata, this ticket is about loaddata. The problem with loaddata is that it would be very difficult to incrementally read the file.

[Berker Peksag]

json.Deserializer now uses six.reraise (see #18003). I think there are two questions that needs to be answered here:

1. Should DeserializationError wrap MemoryError? If not, we need a patch to catch MemoryError explicitly.
2. If yes, the ticket's title and description should be updated.

See also #12007.

[Jacob Walls]

​PR on the assumption we don't want to wrap MemoryError. If we do want to wrap it, we're already chaining the exception with raise from so we can just close the ticket.

[Mariusz Felisiak]

MemoryError is not wrapped anymore because we no longer use StringIO or BytesIO, and MemoryError is raised by .read() (see 7d0f8831922535502569a5dda989dde339b4e491):

  File "django/django/core/serializers/json.py", line 65, in Deserializer
    stream_or_string = stream_or_string.read()
MemoryError: Problem installing fixture 'mydump.json'

[Claude Paroz]

I'm not so sure that MemoryError is only triggered by read(), but let's see how it goes and if the issue happens again in the future.