Opened 9 years ago

Closed 5 years ago

Last modified 5 years ago

#6773 closed Cleanup/optimization (needsinfo)

Confusing error when ForeignKey lookup fails while serializing

Reported by: Kenneth Arnold Owned by:
Component: Core (Serialization) Version: 1.3
Severity: Normal Keywords: dumpdata exception
Cc: kenneth.arnold@…, e.robillard@…, moya@…, Gabriel Hurley, Ryan Witt, ethanp Triage Stage: Design decision needed
Has patch: no Needs documentation: yes
Needs tests: yes Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I got a confusing error when running manage.py dumpdata:

Error: Unable to serialize database:

Turns out that a ForeignKey had been marked blank=True but not null=True.

The error message gave no hint as to even which model to look in; the real exception lay hidden behind two levels of error handlers. Maybe serializers should pass on exceptions?

Attachments (1)

find_serialization_errors.py (4.2 KB) - added by Ryan Witt 6 years ago.
management command to find serialization errors

Download all attachments as: .zip

Change History (21)

comment:1 Changed 8 years ago by Jeff Anderson

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Triage Stage: UnreviewedDesign decision needed

comment:2 Changed 8 years ago by Kenneth Arnold

Cc: kenneth.arnold@… added

comment:3 Changed 8 years ago by Etienne Robillard

Cc: e.robillard@… added

You can use --traceback to debug this:

$ bin/debug.sh manage dumpdata -v 2 --traceback foo
/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.2-py2.6-linux-i686.egg/M
ySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
/home/erob/foo/lib/foo/captcha/models.py:8: DeprecationWarni
ng: the md5 module is deprecated; use hashlib instead
  import md5
Traceback (most recent call last):
  File "/home/erob/foo/bin/manage.py", line 7, in <module>
    execute_manager(settings_module)
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 340
, in execute_manager
    utility.execute()
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 295
, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/erob/django.readonly/django/core/management/base.py", line 195, in
 run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/erob/django.readonly/django/core/management/base.py", line 222, in    execute
    output = self.handle(*args, **options)
  File "/home/erob/django.readonly/django/core/management/commands/dumpdata.py",    line 48, in handle
    return serializers.serialize(format, objects, indent=indent)
  File "/home/erob/django.readonly/django/core/serializers/__init__.py", line 86   , in serialize
    s.serialize(queryset, **options)
  File "/home/erob/django.readonly/django/core/serializers/base.py", line 47, in    serialize
    self.handle_fk_field(obj, field)
  File "/home/erob/django.readonly/django/core/serializers/python.py", line 41,    in handle_fk_field
    related = getattr(obj, field.name)
  File "/home/erob/django.readonly/django/db/models/fields/related.py", line 248   , in __get__
    rel_obj = QuerySet(self.field.rel.to).get(**params)
  File "/home/erob/django.readonly/django/db/models/query.py", line 309, in get
    % self.model._meta.object_name)
foo.models.DoesNotExist: Foo matching query does not exist.

Afaik, a simple print statement or two in serializers/python.py would not hurt much and
would help identifying buggy code.

comment:4 Changed 7 years ago by anonymous

Cc: moya@… added; kenneth.arnold@… e.robillard@… removed

comment:5 Changed 7 years ago by anonymous

Cc: kenneth.arnold@… e.robillard@… added

comment:6 Changed 7 years ago by ajitomatix

Version: SVN1.0

Even something as simple as the following is helpful to find out why the exception is thrown.

raise self.model.DoesNotExist("%s matching query does not exist. args were %s. Lookup parameters were %s"
                    % (self.model._meta.object_name, args, kwargs))

comment:7 Changed 7 years ago by ajitomatix

Version: 1.0SVN

comment:8 Changed 6 years ago by Gabriel Hurley

Cc: Gabriel Hurley added

comment:9 Changed 6 years ago by Ryan Witt

Cc: Ryan Witt added

comment:10 Changed 6 years ago by googol

Owner: changed from nobody to googol
Status: newassigned

or add even more valuable information - like the table name:

raise self.model.DoesNotExist("%s matching query does not exist. Table name: %s. Args were %s. Lookup parameters were %s"

% (self.model._meta.object_name, self.model._meta.db_table, args, kwargs))

comment:11 Changed 6 years ago by googol

Owner: googol deleted
Status: assignednew

Changed 6 years ago by Ryan Witt

management command to find serialization errors

comment:12 Changed 6 years ago by Ryan Witt

Hopefully the command above will help people track down the offending objects. Should work with Django 1.1-1.2.3 at least. There's a gist of it at https://gist.github.com/724466 if people feel like forking it, but probably should re-attach improved versions here.

See http://docs.djangoproject.com/en/dev/howto/custom-management-commands/ for using the script if you're not familiar with custom commands.

comment:13 Changed 6 years ago by anonymous

milestone: 1.3
Needs documentation: set
Needs tests: set
Triage Stage: Design decision neededFixed on a branch
Version: SVN1.2-alpha

comment:14 Changed 5 years ago by Julien Phalip

Severity: Normal
Type: Cleanup/optimization

comment:15 Changed 5 years ago by Dan Loewenherz

Easy pickings: unset
milestone: 1.3
UI/UX: unset

What's the status on this ticket? I'm removing the 1.3 milestone since 1.3 has already been released.

What documentation is needed for this? Perhaps a paragraph explaining the issue?

comment:16 Changed 5 years ago by Russell Keith-Magee

Resolution: needsinfo
Status: newclosed
Triage Stage: Fixed on a branchDesign decision needed

The answers you seek can be partially found by reading the ticket history.

The ticket *was* DDN until it was marked "Fixed on a branch" by an anonymous user (who also marked it for milestone 1.3). Since the anonymous person didn't give any explanation, you can probably assume that that update was spam. That means the ticket is waiting on a design decision.

A patch has been provided; that patch isn't a diff against trunk, and it doesn't include tests. It also introduces a new management command, but includes no documentation for that management command. Assuming that the patch is the right fix, the patch isn't ready for trunk because of these missing elements.

However, I'm not convinced the patch is the right approach. We already have a "find the errors" command -- it's called validate. I don't see the use case for an additional management command to catch serialization errors. In any case, blank=True without null=True shouldn't pose a serialization problem, but the reporter doesn't provide a clear test case to validate the problem they describe.

Given that this problem was reported 3 years ago, there have been many improvements in serialization error reporting since that time, and many edge cases caught in serialization since then, I'm going to close this needsinfo. If someone can verify that this is still a problem, feel free to reopen, providing a specific reproduction test case.

comment:17 Changed 5 years ago by littke

Even though I'm unable to provide a specific reproduction test case, I can report that I'm having this issue in Django 1.3 with validate returning 0 errors while the dumpdata reports "Error: Unable to serialize database: Package matching query does not exist."

The --traceback command does not help much.

In my case, I believe this occured because of a row with a foreignkey pointing to a parent that was deleted.

comment:18 Changed 5 years ago by ethanp

Cc: ethanp added

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

comment:19 in reply to:  18 Changed 5 years ago by anonymous

Replying to ethanp:

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

Thank you!
a simple, bare, <null> (without the brackets) in place of a 0 for null foreign key properties in an import json fixture will also take care of the problem.

comment:20 Changed 5 years ago by nathank@…

Version: 1.2-alpha1.3

still a problem in 1.3.1.

Note: See TracTickets for help on using tickets.
Back to Top