Opened 16 years ago

Closed 13 years ago

Last modified 12 years ago

#6773 closed Cleanup/optimization (needsinfo)

Confusing error when ForeignKey lookup fails while serializing

Reported by: Kenneth Arnold Owned by:
Component: Core (Serialization) Version: 1.3
Severity: Normal Keywords: dumpdata exception
Cc: kenneth.arnold@…, e.robillard@…, moya@…, Gabriel Hurley, Ryan Witt, ethanp Triage Stage: Design decision needed
Has patch: no Needs documentation: yes
Needs tests: yes Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I got a confusing error when running manage.py dumpdata:

Error: Unable to serialize database:

Turns out that a ForeignKey had been marked blank=True but not null=True.

The error message gave no hint as to even which model to look in; the real exception lay hidden behind two levels of error handlers. Maybe serializers should pass on exceptions?

Attachments (1)

find_serialization_errors.py (4.2 KB ) - added by Ryan Witt 13 years ago.
management command to find serialization errors

Download all attachments as: .zip

Change History (21)

comment:1 by Jeff Anderson, 16 years ago

Triage Stage: UnreviewedDesign decision needed

comment:2 by Kenneth Arnold, 16 years ago

Cc: kenneth.arnold@… added

comment:3 by Etienne Robillard, 15 years ago

Cc: e.robillard@… added

You can use --traceback to debug this:

$ bin/debug.sh manage dumpdata -v 2 --traceback foo
/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.2-py2.6-linux-i686.egg/M
ySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
/home/erob/foo/lib/foo/captcha/models.py:8: DeprecationWarni
ng: the md5 module is deprecated; use hashlib instead
  import md5
Traceback (most recent call last):
  File "/home/erob/foo/bin/manage.py", line 7, in <module>
    execute_manager(settings_module)
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 340
, in execute_manager
    utility.execute()
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 295
, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/erob/django.readonly/django/core/management/base.py", line 195, in
 run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/erob/django.readonly/django/core/management/base.py", line 222, in    execute
    output = self.handle(*args, **options)
  File "/home/erob/django.readonly/django/core/management/commands/dumpdata.py",    line 48, in handle
    return serializers.serialize(format, objects, indent=indent)
  File "/home/erob/django.readonly/django/core/serializers/__init__.py", line 86   , in serialize
    s.serialize(queryset, **options)
  File "/home/erob/django.readonly/django/core/serializers/base.py", line 47, in    serialize
    self.handle_fk_field(obj, field)
  File "/home/erob/django.readonly/django/core/serializers/python.py", line 41,    in handle_fk_field
    related = getattr(obj, field.name)
  File "/home/erob/django.readonly/django/db/models/fields/related.py", line 248   , in __get__
    rel_obj = QuerySet(self.field.rel.to).get(**params)
  File "/home/erob/django.readonly/django/db/models/query.py", line 309, in get
    % self.model._meta.object_name)
foo.models.DoesNotExist: Foo matching query does not exist.

Afaik, a simple print statement or two in serializers/python.py would not hurt much and
would help identifying buggy code.

comment:4 by anonymous, 15 years ago

Cc: moya@… added; kenneth.arnold@… e.robillard@… removed

comment:5 by anonymous, 15 years ago

Cc: kenneth.arnold@… e.robillard@… added

comment:6 by ajitomatix, 15 years ago

Version: SVN1.0

Even something as simple as the following is helpful to find out why the exception is thrown.

raise self.model.DoesNotExist("%s matching query does not exist. args were %s. Lookup parameters were %s"
                    % (self.model._meta.object_name, args, kwargs))

comment:7 by ajitomatix, 15 years ago

Version: 1.0SVN

comment:8 by Gabriel Hurley, 14 years ago

Cc: Gabriel Hurley added

comment:9 by Ryan Witt, 14 years ago

Cc: Ryan Witt added

comment:10 by googol, 14 years ago

Owner: changed from nobody to googol
Status: newassigned

or add even more valuable information - like the table name:

raise self.model.DoesNotExist("%s matching query does not exist. Table name: %s. Args were %s. Lookup parameters were %s"

% (self.model._meta.object_name, self.model._meta.db_table, args, kwargs))

comment:11 by googol, 14 years ago

Owner: googol removed
Status: assignednew

by Ryan Witt, 13 years ago

management command to find serialization errors

comment:12 by Ryan Witt, 13 years ago

Hopefully the command above will help people track down the offending objects. Should work with Django 1.1-1.2.3 at least. There's a gist of it at https://gist.github.com/724466 if people feel like forking it, but probably should re-attach improved versions here.

See http://docs.djangoproject.com/en/dev/howto/custom-management-commands/ for using the script if you're not familiar with custom commands.

comment:13 by anonymous, 13 years ago

milestone: 1.3
Needs documentation: set
Needs tests: set
Triage Stage: Design decision neededFixed on a branch
Version: SVN1.2-alpha

comment:14 by Julien Phalip, 13 years ago

Severity: Normal
Type: Cleanup/optimization

comment:15 by Dan Loewenherz, 13 years ago

Easy pickings: unset
milestone: 1.3
UI/UX: unset

What's the status on this ticket? I'm removing the 1.3 milestone since 1.3 has already been released.

What documentation is needed for this? Perhaps a paragraph explaining the issue?

comment:16 by Russell Keith-Magee, 13 years ago

Resolution: needsinfo
Status: newclosed
Triage Stage: Fixed on a branchDesign decision needed

The answers you seek can be partially found by reading the ticket history.

The ticket *was* DDN until it was marked "Fixed on a branch" by an anonymous user (who also marked it for milestone 1.3). Since the anonymous person didn't give any explanation, you can probably assume that that update was spam. That means the ticket is waiting on a design decision.

A patch has been provided; that patch isn't a diff against trunk, and it doesn't include tests. It also introduces a new management command, but includes no documentation for that management command. Assuming that the patch is the right fix, the patch isn't ready for trunk because of these missing elements.

However, I'm not convinced the patch is the right approach. We already have a "find the errors" command -- it's called validate. I don't see the use case for an additional management command to catch serialization errors. In any case, blank=True without null=True shouldn't pose a serialization problem, but the reporter doesn't provide a clear test case to validate the problem they describe.

Given that this problem was reported 3 years ago, there have been many improvements in serialization error reporting since that time, and many edge cases caught in serialization since then, I'm going to close this needsinfo. If someone can verify that this is still a problem, feel free to reopen, providing a specific reproduction test case.

comment:17 by littke, 13 years ago

Even though I'm unable to provide a specific reproduction test case, I can report that I'm having this issue in Django 1.3 with validate returning 0 errors while the dumpdata reports "Error: Unable to serialize database: Package matching query does not exist."

The --traceback command does not help much.

In my case, I believe this occured because of a row with a foreignkey pointing to a parent that was deleted.

comment:18 by ethanp, 13 years ago

Cc: ethanp added

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

in reply to:  18 comment:19 by anonymous, 13 years ago

Replying to ethanp:

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

Thank you!
a simple, bare, <null> (without the brackets) in place of a 0 for null foreign key properties in an import json fixture will also take care of the problem.

comment:20 by nathank@…, 12 years ago

Version: 1.2-alpha1.3

still a problem in 1.3.1.

Note: See TracTickets for help on using tickets.
Back to Top