Opened 10 years ago
Closed 7 years ago
#24607 closed Bug (fixed)
Serialization (and deserialization) of MTI models doesn't work with natural keys
Reported by: | Denys Duchier | Owned by: | |
---|---|---|---|
Component: | Core (Serialization) | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Ready for checkin | |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
I am using Django1.8. The exact error message is:
ValueError: Problem installing fixture '/home/denys/src/GSM/tmp/bug/person.json': "<Person: >" needs to have a value for field "user" before this many-to-many relationship can be used.
I have created a very small example to reproduce the problem; it is available on github at:
https://github.com/denys-duchier/django-loaddata-bug.git
to create the database, migrate, populate the database, then dumpdata
, execute:
step1.bash
to recreate the database, migrate, and attempt the loaddata
, execute:
step2.bash
that is when the error occurs.
Change History (20)
comment:1 by , 10 years ago
Component: | Uncategorized → Core (Serialization) |
---|---|
Triage Stage: | Unreviewed → Accepted |
Type: | Uncategorized → Bug |
comment:3 by , 10 years ago
I am thinking that when we have use_natural_foreign_keys
and the pk
is a relation, we want to recurse.
comment:6 by , 10 years ago
Has patch: | set |
---|
comment:7 by , 9 years ago
Patch needs improvement: | set |
---|
Waiting for a response from the reporter on the pull request, but it seems to me the issue is that use_natural_primary_keys
is not compatible with serializing multi-table inheritance models since the child models require the primary key to be linked to their parent. I'm not sure that adding a lot of complexity to essential "undo" the option for the affected models is worth it. Could we simply document the limitation?
comment:8 by , 8 years ago
As it stands right now, use_natural_primary_keys
doesn't work with multi-table inheritance with or without use_natural_foreign_keys
-- but there's no fundamental reason stopping this and the most recent patch at least allows it to work if both use_natural_primary_keys
and use_natural_foreign_keys
are enabled.
The patch changes serialization requires use_natural_primary_keys
and use_natural_foreign_keys
to both be set in the case of multi-table inheritance. Possible options where use_natural_primary_keys
is set but use_natural_foreign_keys
isn't:
use_natural_primary_keys
impliesuse_natural_foreign_keys
- Always (this is what the patch did)
- If there are any multi-table inheritance models defined that have a natural key
- Only for references to models that are involved in any multi-table inheritance
- May be a little surprising if you add an inherited model and it changes the dump for the parent model changes
use_natural_primary_keys
does not implyuse_natural_foreign_keys
- dumps may break if load/dump order is not identical as the real PKs may not match (ie is likely to break)
- If there are any multi-table inheritance models with natural keys then a warning can be emitted
use_natural_primary_keys
is ignored for models that are involved in any multi-table inheritance and have a natural_key defined- May be a little surprising if you add an inherited model and it changes the dump for the parent model changes
Option 2 involves the least change and with the warning follows the principle of least surprise. It is an improvement over the current situation and is as backwards-compatible as possible.
If the patch was bought up to date with the latest django version, and it was changed to use option 2 (so use_natural_primary_keys
did not turn on use_natural_foreign_keys
but gave a warning if a likely problem was detected) then would this be acceptable?
(As an aside: if use_natural_primary_keys
is set but use_natural_foreign_keys
isn't then even for tables without multi-table inheritance it would seem that there is a sensitivity to the load/dump order in the referenced/referencing tables. The documentation at https://docs.djangoproject.com/en/dev/topics/serialization/ doesn't mention this anywhere)
comment:9 by , 8 years ago
Summary: | dumpdata (with natural keys) / loaddata failure for model inheriting from User → Serialization (and deserialization) of MTI models doesn't work with natural keys |
---|---|
Version: | 1.8 → master |
comment:10 by , 8 years ago
comment:11 by , 8 years ago
Patch needs improvement: | unset |
---|
comment:15 by , 8 years ago
Has patch: | unset |
---|---|
Resolution: | fixed |
Status: | closed → new |
Reopening per discussion on #27742.
comment:16 by , 8 years ago
new PR: https://github.com/django/django/pull/8370
I slightly reworked @jpmelos earlier PR #7231 to hopefully avoid the problem with spurious migrations. When a relation with parent_link=True
is used as a pk then, when serializing with natural primary keys, the field must be serialized, even when marked with serialize=False
.
comment:17 by , 7 years ago
Has patch: | set |
---|---|
Owner: | changed from | to
Status: | new → assigned |
I'm reviewing the latest PR on https://github.com/django/django/pull/8370
comment:18 by , 7 years ago
Owner: | removed |
---|---|
Status: | assigned → new |
Triage Stage: | Accepted → Ready for checkin |
comment:19 by , 7 years ago
posting into here for convenience:
For people that need something working now, I've worked this PR into a standalone serializer. It works for our use case, however I can't guarantee it will work in all circumstances.
https://gist.github.com/cybojenix/2476434c513170d2f267cb588a8d645b
The issue seems to be caused by the fact that
--natural-primary
causes thePerson
's 'id' to be omitted in the fixture dump (sinceAbstractBaseUser
definesnatural_key()
), but without it, there's no way to connectPerson
instances to the correspondingUser
.