Opened 10 months ago

Closed 8 months ago

#28972 closed Bug (duplicate)

`RelatedObjectDoesNotExist` during `loaddata` of models with multi-table inheritance

Reported by: Kal Sze Owned by: nobody
Component: Core (Serialization) Version: 2.0
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Kal Sze)

Software versions

  • Python 3.6.3
  • Django 2.0
  • django-model-utils 3.1.1 (from git)
  • PostgreSQL 9.4
  • psycopg2 2.7.3.2

I have a licenses app, with these models:

  • BaseLicense, a concrete model, with a ForeignKey to settings.AUTH_USER_MODEL.
  • CorelDrawLicense, which inherits from BaseLicense through multi-table inheritance
  • AdobePhotoshopLicense, which inherits from BaseLicense through multi-table inheritance

I created an instance of AdobePhotoshopLicense, then tried to do manage.py dumpdata and manage.py loaddata.

dumpdata "worked" - there was no error.

However, loaddata failed with this traceback:

Traceback (most recent call last):
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/json.py", line 69, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/python.py", line 170, in Deserializer
    obj = base.build_instance(Model, data, using)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/base.py", line 225, in build_instance
    natural_key = obj.natural_key()
  File "/home/kal/git/nk_test/licenses/models.py", line 33, in natural_key
    return (self.user.natural_key(), self.license_key)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 175, in __get__
    "%s has no %s." % (self.field.model.__name__, self.field.name)
django.db.models.fields.related_descriptors.RelatedObjectDoesNotExist: BaseLicense has no user.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 113, in loaddata
    self.load_label(fixture_label)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 168, in load_label
    for obj in objects:
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/json.py", line 73, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/home/kal/git/nk_test/./fixtures/dumpdata.json':

Here are the exact steps:

  1. Create model instances in management shell
    >>> from django.contrib.auth import get_user_model
    >>> User = get_user_model()
    >>> kal = User.objects.create_user('kal', 'kal@foobar.com', 'qwer1234')
    >>> from licenses.models import AdobePhotoshopLicense
    >>> l = AdobePhotoshopLicense.objects.create(user=kal, license_key='12345')
    
  1. Dump the data into a fixtures json
    $ python manage.py  dumpdata -o fixtures.json --format json --natural-foreign --natural-primary auth licenses
    
  1. Edit settings.py to switch to an empty database
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            #'NAME': 'nk_test',
            'NAME': 'nk_test_2',
            'USER': 'kal',
            'HOST': '',
            'PORT': 5432,
        }
    }
    
  1. Run migrations on the new empty database
    $ python manage.py migrate
    
  1. Try to load the data from fixtures.json
    $ python manage.py loaddata --format json fixtures.json
    

I have attached the models.py and fixtures.json files to the ticket.

I know that I don't (and I can't) enforce the natural key uniqueness at the database level. However, due to my application logic, I know that the user+license_key combination is effectively unique for any one of the child license models. And this is specifically allowed, according to the documentation:

[U]niqueness doesn’t need to be enforced at the database level. If you are certain that a set of fields will be effectively unique, you can still use those fields as a natural key.

Also note that this bug doesn't have to do with my use of the custom InheritanceManager. I could let the BaseLicense model use the default manager and still get the same bug.

Attachments (2)

models.py (1.4 KB) - added by Kal Sze 10 months ago.
models.py
fixtures.json (4.9 KB) - added by Kal Sze 10 months ago.
fixtures.json, reformatted for readability

Download all attachments as: .zip

Change History (10)

Changed 10 months ago by Kal Sze

Attachment: models.py added

models.py

Changed 10 months ago by Kal Sze

Attachment: fixtures.json added

fixtures.json, reformatted for readability

comment:1 Changed 10 months ago by Kal Sze

Description: modified (diff)

comment:2 Changed 10 months ago by Kal Sze

Description: modified (diff)

comment:3 Changed 10 months ago by Kal Sze

Description: modified (diff)

comment:4 Changed 10 months ago by Tim Graham

It may be a duplicate of #24607. Could you test the patch there?

comment:5 in reply to:  4 Changed 10 months ago by Kal Sze

Resolution: duplicate
Status: newclosed

Replying to Tim Graham:

It may be a duplicate of #24607. Could you test the patch there?

PR 8370 fixes it.

I did see #24607, but I didn't think it was the same problem because the symptoms were a bit different: I got a RelatedObjectDoesNotExist whereas as the original ticket got a ValueError.

comment:6 Changed 10 months ago by Kal Sze

Resolution: duplicate
Status: closednew

Actually there is still a bit of a problem.

Even though serialization and deserialization "work", I think the result still misses the point of the natural key.

If I look at the fixtures being dumped out, the parent model is still referred to by pk:

[...
{
	"model": "licenses.baselicense",
	"pk": 1,
	"fields": {
		"user": ["kal"]
	}
},
{
	"model": "licenses.adobephotoshoplicense",
	"fields": {
		"baselicense_ptr": 1,
		"license_key": "12345"
	}
}]

Whereas I expect it to be just something like:

{
	"model": "licenses.adobephotoshoplicense",
	"fields": {
		"user": ["kal"],
		"license_key": "12345"
	}
}
Last edited 10 months ago by Kal Sze (previous) (diff)

comment:7 Changed 10 months ago by Tim Graham

Is that a separate issue or a defect in the patch for #24607? If the latter, I'd point out the issue there rather than opening a different ticket.

comment:8 Changed 8 months ago by Tim Graham

Resolution: duplicate
Status: newclosed

I don't think the output you proposed could be deserialized properly. If baselicense_ptr isn't included in the adobephotoshoplicense object, there's no way to link adobephotoshoplicense to its baselicense object since user doens't uniquely identify a baselicense.

Note: See TracTickets for help on using tickets.
Back to Top