Make loaddata error messages less cryptic

[bigfudge]

At present, error messages when running loaddata don't include any clue about the part of the serialised data which caused the problem.

In the Deserialiser function (in core.serializers.json.py) there is a try/except block on line 40 to 47 which could be amended to aid the user:

def Deserializer(stream_or_string, **options):
    """
    Deserialize a stream or string of JSON data.
    """
    if isinstance(stream_or_string, basestring):
        stream = StringIO(stream_or_string)
    else:
        stream = stream_or_string
    try:
        for obj in PythonDeserializer(simplejson.load(stream), **options):
            yield obj
    except GeneratorExit:
        raise
    except Exception, e:
        # Map to deserializer error
        raise DeserializationError(e)

If raise DeserializationError(e) were changed to raise DeserializationError(e, obj) then the function would return a tuple containing the current error message (which isn't that informative) and the object which causes the error.

This has been a big issue for me recently in migrating a pilot installation which used sqlite over to postgres. There were a number of data integrity issues which using postgres highlighted, but it was only by applying this change that it became possible to identify the records which were causing the problem.

[Claude Paroz]

Better error messages are always welcome. However, you mention integrity issues, while the fix you suggest is at deserialization stage. Maybe some actual example could help here?

[Carl Meyer]

Closing needsinfo pending clarification of whether the better error messages are really needed at deserialization or database-insert, and an example clearly demonstrating the problem that the proposed fix would ameliorate.

[no]

If there is, for example, an initialiser for auth.Permission with a content_type that does not exist, the DeserializationError is raised:

django.core.serializers.base.DeserializationError: Problem installing fixture 'testapp/initial_data/testing_data.json': ContentType matching query does not exist.

This is a problem with big files, since it doesn't tell you where in the file the error occurred. (Reproduced on 1.7c3)

[no]

Proposed fix: ​https://github.com/django/django/pull/3098

[no]

I've add a simple patch which tests that an error message is raised, and that the message contains the offending model name, it's PK, and what the offending field's value is.

Let me know if there's anything else I can add.

[Tim Graham]

I left comments for improvement on the PR. Please uncheck "Patch needs improvement" when you update it, thanks.

[no]

New PR ​https://github.com/django/django/pull/3223

I left some questions I need some feedback on.

[Tim Graham]

Patch needs a rebase.

[no]

Rebased, and responded to question.

[Tim Graham]

Don't forget to uncheck "Patch needs improvement" so the ticket appears in the review queue.

[Tim Graham]

Tests need to fixed on Python 3.

[no]

Fixed python 3 error: it was due to python3 "fixing" variables escaping generator expressions.

[Tim Graham]

Left some more comments.

[no]

Rebased and updated patch.

[Tim Graham <timograham@…>]

In 727e40c8:

Fixed #19820 -- Added more helpful error messages to Python deserializer.