`RelatedObjectDoesNotExist` during `loaddata` of models with multi-table inheritance

[Kal Sze]

Software versions

• Python 3.6.3
• Django 2.0
• django-model-utils 3.1.1 (from git)
• PostgreSQL 9.4
• psycopg2 2.7.3.2

I have a licenses app, with these models:

• BaseLicense, a concrete model, with a ForeignKey to settings.AUTH_USER_MODEL.
• CorelDrawLicense, which inherits from BaseLicense through multi-table inheritance
• AdobePhotoshopLicense, which inherits from BaseLicense through multi-table inheritance

I created an instance of AdobePhotoshopLicense, then tried to do manage.py dumpdata and manage.py loaddata.

dumpdata "worked" - there was no error.

However, loaddata failed with this traceback:

Traceback (most recent call last):
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/json.py", line 69, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/python.py", line 170, in Deserializer
    obj = base.build_instance(Model, data, using)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/base.py", line 225, in build_instance
    natural_key = obj.natural_key()
  File "/home/kal/git/nk_test/licenses/models.py", line 33, in natural_key
    return (self.user.natural_key(), self.license_key)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 175, in __get__
    "%s has no %s." % (self.field.model.__name__, self.field.name)
django.db.models.fields.related_descriptors.RelatedObjectDoesNotExist: BaseLicense has no user.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 113, in loaddata
    self.load_label(fixture_label)
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 168, in load_label
    for obj in objects:
  File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-packages/django/core/serializers/json.py", line 73, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/home/kal/git/nk_test/./fixtures/dumpdata.json':

Here are the exact steps:

1. Create model instances in management shell

   >>> from django.contrib.auth import get_user_model
   >>> User = get_user_model()
   >>> kal = User.objects.create_user('kal', 'kal@foobar.com', 'qwer1234')
   >>> from licenses.models import AdobePhotoshopLicense
   >>> l = AdobePhotoshopLicense.objects.create(user=kal, license_key='12345')
   

2. Dump the data into a fixtures json

   $ python manage.py  dumpdata -o fixtures.json --format json --natural-foreign --natural-primary auth licenses
   

3. Edit settings.py to switch to an empty database

   DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.postgresql',
           #'NAME': 'nk_test',
           'NAME': 'nk_test_2',
           'USER': 'kal',
           'HOST': '',
           'PORT': 5432,
       }
   }
   

4. Run migrations on the new empty database

   $ python manage.py migrate
   

5. Try to load the data from fixtures.json

   $ python manage.py loaddata --format json fixtures.json
   

I have attached the models.py and fixtures.json files to the ticket.

I know that I don't (and I can't) enforce the natural key uniqueness at the database level. However, due to my application logic, I know that the user+license_key combination is effectively unique for any one of the child license models. And this is specifically allowed, ​according to the documentation:

[U]niqueness doesn’t need to be enforced at the database level. If you are certain that a set of fields will be effectively unique, you can still use those fields as a natural key.

Also note that this bug doesn't have to do with my use of the custom InheritanceManager. I could let the BaseLicense model use the default manager and still get the same bug.

[Kal Sze]

models.py

[Kal Sze]

fixtures.json, reformatted for readability

[Tim Graham]

It may be a duplicate of #24607. Could you test the patch there?

[Kal Sze]

Replying to Tim Graham:

It may be a duplicate of #24607. Could you test the patch there?

​PR 8370 fixes it.

I did see #24607, but I didn't think it was the same problem because the symptoms were a bit different: I got a RelatedObjectDoesNotExist whereas as the original ticket got a ValueError.

[Kal Sze]

Actually there is still a bit of a problem.

Even though serialization and deserialization "work", I think the result still misses the point of the natural key.

If I look at the fixtures being dumped out, the parent model is still referred to by pk:

[...
{
	"model": "licenses.baselicense",
	"pk": 1,
	"fields": {
		"user": ["kal"]
	}
},
{
	"model": "licenses.adobephotoshoplicense",
	"fields": {
		"baselicense_ptr": 1,
		"license_key": "12345"
	}
}]

Whereas I expect it to be just something like:

{
	"model": "licenses.adobephotoshoplicense",
	"fields": {
		"user": ["kal"],
		"license_key": "12345"
	}
}

[Tim Graham]

Is that a separate issue or a defect in the patch for #24607? If the latter, I'd point out the issue there rather than opening a different ticket.

[Tim Graham]

I don't think the output you proposed could be deserialized properly. If baselicense_ptr isn't included in the adobephotoshoplicense object, there's no way to link adobephotoshoplicense to its baselicense object since user doens't uniquely identify a baselicense.