Opened 7 years ago

Closed 4 years ago

Last modified 4 years ago

#6773 closed Cleanup/optimization (needsinfo)

Confusing error when ForeignKey lookup fails while serializing

Reported by: kcarnold Owned by:
Component: Core (Serialization) Version: 1.3
Severity: Normal Keywords: dumpdata exception
Cc: kenneth.arnold@…, e.robillard@…, moya@…, gabrielhurley, onecreativenerd, ethanp Triage Stage: Design decision needed
Has patch: no Needs documentation: yes
Needs tests: yes Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I got a confusing error when running manage.py dumpdata:

Error: Unable to serialize database:

Turns out that a ForeignKey had been marked blank=True but not null=True.

The error message gave no hint as to even which model to look in; the real exception lay hidden behind two levels of error handlers. Maybe serializers should pass on exceptions?

Attachments (1)

find_serialization_errors.py (4.2 KB) - added by onecreativenerd 5 years ago.
management command to find serialization errors

Download all attachments as: .zip

Change History (21)

comment:1 Changed 7 years ago by programmerq

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Design decision needed

comment:2 Changed 7 years ago by kcarnold

  • Cc kenneth.arnold@… added

comment:3 Changed 7 years ago by erob

  • Cc e.robillard@… added

You can use --traceback to debug this:

$ bin/debug.sh manage dumpdata -v 2 --traceback foo
/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.2-py2.6-linux-i686.egg/M
ySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
/home/erob/foo/lib/foo/captcha/models.py:8: DeprecationWarni
ng: the md5 module is deprecated; use hashlib instead
  import md5
Traceback (most recent call last):
  File "/home/erob/foo/bin/manage.py", line 7, in <module>
    execute_manager(settings_module)
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 340
, in execute_manager
    utility.execute()
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 295
, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/erob/django.readonly/django/core/management/base.py", line 195, in
 run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/erob/django.readonly/django/core/management/base.py", line 222, in    execute
    output = self.handle(*args, **options)
  File "/home/erob/django.readonly/django/core/management/commands/dumpdata.py",    line 48, in handle
    return serializers.serialize(format, objects, indent=indent)
  File "/home/erob/django.readonly/django/core/serializers/__init__.py", line 86   , in serialize
    s.serialize(queryset, **options)
  File "/home/erob/django.readonly/django/core/serializers/base.py", line 47, in    serialize
    self.handle_fk_field(obj, field)
  File "/home/erob/django.readonly/django/core/serializers/python.py", line 41,    in handle_fk_field
    related = getattr(obj, field.name)
  File "/home/erob/django.readonly/django/db/models/fields/related.py", line 248   , in __get__
    rel_obj = QuerySet(self.field.rel.to).get(**params)
  File "/home/erob/django.readonly/django/db/models/query.py", line 309, in get
    % self.model._meta.object_name)
foo.models.DoesNotExist: Foo matching query does not exist.

Afaik, a simple print statement or two in serializers/python.py would not hurt much and
would help identifying buggy code.

comment:4 Changed 6 years ago by anonymous

  • Cc moya@… added; kenneth.arnold@… e.robillard@… removed

comment:5 Changed 6 years ago by anonymous

  • Cc kenneth.arnold@… e.robillard@… added

comment:6 Changed 6 years ago by ajitomatix

  • Version changed from SVN to 1.0

Even something as simple as the following is helpful to find out why the exception is thrown.

raise self.model.DoesNotExist("%s matching query does not exist. args were %s. Lookup parameters were %s"
                    % (self.model._meta.object_name, args, kwargs))

comment:7 Changed 6 years ago by ajitomatix

  • Version changed from 1.0 to SVN

comment:8 Changed 5 years ago by gabrielhurley

  • Cc gabrielhurley added

comment:9 Changed 5 years ago by onecreativenerd

  • Cc onecreativenerd added

comment:10 Changed 5 years ago by googol

  • Owner changed from nobody to googol
  • Status changed from new to assigned

or add even more valuable information - like the table name:

raise self.model.DoesNotExist("%s matching query does not exist. Table name: %s. Args were %s. Lookup parameters were %s"

% (self.model._meta.object_name, self.model._meta.db_table, args, kwargs))

comment:11 Changed 5 years ago by googol

  • Owner googol deleted
  • Status changed from assigned to new

Changed 5 years ago by onecreativenerd

management command to find serialization errors

comment:12 Changed 5 years ago by onecreativenerd

Hopefully the command above will help people track down the offending objects. Should work with Django 1.1-1.2.3 at least. There's a gist of it at https://gist.github.com/724466 if people feel like forking it, but probably should re-attach improved versions here.

See http://docs.djangoproject.com/en/dev/howto/custom-management-commands/ for using the script if you're not familiar with custom commands.

comment:13 Changed 5 years ago by anonymous

  • milestone set to 1.3
  • Needs documentation set
  • Needs tests set
  • Triage Stage changed from Design decision needed to Fixed on a branch
  • Version changed from SVN to 1.2-alpha

comment:14 Changed 4 years ago by julien

  • Severity set to Normal
  • Type set to Cleanup/optimization

comment:15 Changed 4 years ago by dloewenherz

  • Easy pickings unset
  • milestone 1.3 deleted
  • UI/UX unset

What's the status on this ticket? I'm removing the 1.3 milestone since 1.3 has already been released.

What documentation is needed for this? Perhaps a paragraph explaining the issue?

comment:16 Changed 4 years ago by russellm

  • Resolution set to needsinfo
  • Status changed from new to closed
  • Triage Stage changed from Fixed on a branch to Design decision needed

The answers you seek can be partially found by reading the ticket history.

The ticket *was* DDN until it was marked "Fixed on a branch" by an anonymous user (who also marked it for milestone 1.3). Since the anonymous person didn't give any explanation, you can probably assume that that update was spam. That means the ticket is waiting on a design decision.

A patch has been provided; that patch isn't a diff against trunk, and it doesn't include tests. It also introduces a new management command, but includes no documentation for that management command. Assuming that the patch is the right fix, the patch isn't ready for trunk because of these missing elements.

However, I'm not convinced the patch is the right approach. We already have a "find the errors" command -- it's called validate. I don't see the use case for an additional management command to catch serialization errors. In any case, blank=True without null=True shouldn't pose a serialization problem, but the reporter doesn't provide a clear test case to validate the problem they describe.

Given that this problem was reported 3 years ago, there have been many improvements in serialization error reporting since that time, and many edge cases caught in serialization since then, I'm going to close this needsinfo. If someone can verify that this is still a problem, feel free to reopen, providing a specific reproduction test case.

comment:17 Changed 4 years ago by littke

Even though I'm unable to provide a specific reproduction test case, I can report that I'm having this issue in Django 1.3 with validate returning 0 errors while the dumpdata reports "Error: Unable to serialize database: Package matching query does not exist."

The --traceback command does not help much.

In my case, I believe this occured because of a row with a foreignkey pointing to a parent that was deleted.

comment:18 follow-up: Changed 4 years ago by ethanp

  • Cc ethanp added

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

comment:19 in reply to: ↑ 18 Changed 4 years ago by anonymous

Replying to ethanp:

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

Thank you!
a simple, bare, <null> (without the brackets) in place of a 0 for null foreign key properties in an import json fixture will also take care of the problem.

comment:20 Changed 4 years ago by nathank@…

  • Version changed from 1.2-alpha to 1.3

still a problem in 1.3.1.

Note: See TracTickets for help on using tickets.
Back to Top