Opened 7 years ago

Closed 3 years ago

Last modified 21 months ago

#13252 closed New feature (fixed)

Use the natural key instead of the primary key when serializing

Reported by: Chris Beaven Owned by: Malcolm Tredinnick
Component: Core (Serialization) Version: master
Severity: Normal Keywords:
Cc: Tai Lee, chris@…, Chris Spencer, dev@…, charette.s@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

This would allow existing objects to be updated and missing objects to be added across databases (e.g. when the autonumber primary keys may not match).

Attachments (7)

13252.diff (4.0 KB) - added by Chris Beaven 7 years ago.
13252.2.diff (4.8 KB) - added by Chris Beaven 6 years ago.
13252.3.diff (8.8 KB) - added by Chris Beaven 6 years ago.
now with tests/docs
13252-natural-key-serializing-r14992.diff (8.8 KB) - added by Tai Lee 6 years ago.
13252-natural-key-serializing-r14995.diff (39.7 KB) - added by Tai Lee 6 years ago.
13252-natural-key-serializing-r16406.diff (39.1 KB) - added by Claude Paroz 5 years ago.
Updated to apply cleanly on current trunk
13252-natural-key-serializing-r17262.diff (39.7 KB) - added by Claude Paroz 5 years ago.
Updated to current trunk

Download all attachments as: .zip

Change History (36)

Changed 7 years ago by Chris Beaven

Attachment: 13252.diff added

comment:1 Changed 7 years ago by Chris Beaven

Has patch: set
Needs documentation: set
Needs tests: set
Patch needs improvement: unset

This also fixes #11486 by proxy.

comment:2 Changed 7 years ago by Russell Keith-Magee

Patch needs improvement: set
Triage Stage: UnreviewedAccepted

I accept the use case - that you should be able to say "update the existing object that matches the natural key, or create a new object if one doesn't exist". However I'm not convinced that providing a natural key for the PK value is the right way to do this.

Essentially, this approach means specifying the natural key data twice - once in the fields of the object, and once in the primary key value. This strikes me as especially un-DRY.

I haven't given this a lot of thought, but there might be a way of using the natural key without to specify it in the pk attribute. A natural key needs to be unique, so if a model defines a natural key, you shouldn't need to have the PK value specifically (or the natural key value as the primary key) in order to reconstruct the natural key for an object. This does leave the question of when the 'update or create' behavior should be invoked, but I think this will be less prone to error than reproducing the natural key.

Changed 6 years ago by Chris Beaven

Attachment: 13252.2.diff added

comment:3 Changed 6 years ago by Chris Beaven

Patch needs improvement: unset

Changed 6 years ago by Chris Beaven

Attachment: 13252.3.diff added

now with tests/docs

comment:4 Changed 6 years ago by Chris Beaven

Needs documentation: unset
Needs tests: unset

comment:5 Changed 6 years ago by Chris Beaven

Owner: changed from nobody to Malcolm Tredinnick

comment:6 Changed 6 years ago by Paul Winkler

fwiw, patch 3 works for me in initial testing. Really useful feature!

comment:7 Changed 6 years ago by Tai Lee

Cc: Tai Lee added

Changed 6 years ago by Tai Lee

comment:8 Changed 6 years ago by Tai Lee

Updated patch to work on r14992. Appears to work well locally. serializers_regress tests pass (on sqlite at least).

comment:9 Changed 6 years ago by Alex Gaynor

Resolution: fixed
Status: newclosed

Fixed by [14994].

comment:10 Changed 6 years ago by Alex Gaynor

Resolution: fixed
Status: closedreopened

Reopening.

Changed 6 years ago by Tai Lee

comment:11 Changed 6 years ago by Tai Lee

I've just updated the patch after feedback from freakboy3742 on IRC.

The new patch adds --natural-foreign and --natural-primary options to the dumpdata management command, maps the -n and --natural options to --natural-foreign and adds a note in the documentation that the old options are deprecated. This should retain backwards compatibility with anyone who was depending on data dumped with natural keys still containing a pk field for models that support natural keys. It also adds the corresponding use_natural_foreign_keys and use_natural_primary_keys arguments to serializers.serialize() and maps use_natural_keys to use_natural_foreign_keys.

I've left a comment in the source code where I think we should probably raise a deprecation warning (but not actually raised it) if that ends up being the desired behaviour.

One quirk is that the code beginning at line 30 in django/core/serializers/python.py appears to affect the order of fields in serialized output depending on how the data dict is constructed, even though the resulting data dict contains the same data. If I first create the data dict with model and fields items and then conditionally add a pk item (as the old patch did), then the serialized output is different to when I create the data dict with model, pk and fields items. Rather than change a tonne of hard coded tests, I construct the data dict the same way that it would have been before the patch which results in a couple of duplicated lines of code.

comment:12 Changed 5 years ago by Luke Plant

Type: New feature

comment:13 Changed 5 years ago by Luke Plant

Severity: Normal

comment:14 Changed 5 years ago by patchhammer

Easy pickings: unset
Patch needs improvement: set

13252-natural-key-serializing-r14995.diff fails to apply cleanly on to trunk

Changed 5 years ago by Claude Paroz

Updated to apply cleanly on current trunk

comment:15 Changed 5 years ago by Claude Paroz

Patch needs improvement: unset
UI/UX: unset

comment:16 Changed 5 years ago by Chris Beaven

Patch needs improvement: set

The new behaviour needs to be optional to be backwards compatible.

comment:17 Changed 5 years ago by Chris Adams

Cc: chris@… added

comment:18 Changed 5 years ago by Chris Spencer

Cc: Chris Spencer added

Changed 5 years ago by Claude Paroz

Updated to current trunk

comment:19 Changed 5 years ago by Brillgen Developers

Cc: dev@… added

comment:20 Changed 4 years ago by Simon Charette

Cc: charette.s@… added

comment:21 Changed 4 years ago by Tai Lee

I've updated this patch again on a branch at GitHub.

https://github.com/thirstydigital/django/tree/tickets/13252-natural-key-serializing

SmileyChris, can you elaborate on making the new behaviour optional? Unless I'm missing something, it is? The --natural option maps to --natural-foreign (which behaves the same, with a pending deprecation warning) and we now have a new --natural-primary option as well.

I believe I've incorporated all the changes russellm requested and this should be RFC now.

comment:22 Changed 4 years ago by Simon Charette

FWIW all tests pass on python 2.7.3rc1 sqlite3. I guess the backward compatibility or must opt-in issue SmileyChris was referring to is no more with two new flags?

comment:23 Changed 4 years ago by anonymous

I would be very pleased to see this feature enhancement reach the master branch asap 'cause I actually pretty much need it and have write myself some kind of the same..

comment:24 Changed 4 years ago by Tai Lee

Rebased onto master and opened a pull request.

https://github.com/django/django/pull/894

comment:25 Changed 4 years ago by Aymeric Augustin

Status: reopenednew

comment:26 Changed 3 years ago by Tai Lee

Patch needs improvement: unset

Rebased onto master and changes for feedback integrated.

comment:27 Changed 3 years ago by Tim Graham

Patch needs improvement: set

Left a couple more comments on the PR.

comment:28 Changed 3 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In e527c0b6d808cb8e4bedf79ded3dc4ad1a7e17a8:

Fixed #13252 -- Added ability to serialize with natural primary keys.

Added --natural-foreign and --natural-primary options and
deprecated the --natural option to the dumpdata management
command.

Added use_natural_foreign_keys and use_natural_primary_keys
arguments and deprecated the use_natural_keys argument to
django.core.serializers.Serializer.serialize().

Thanks SmileyChris for the suggestion.

comment:29 Changed 21 months ago by Tim Graham <timograham@…>

In c3336e7e4f146fc62272d462288a00f8d78c1f83:

Removed dumpdata --natural option and serializers use_natural_keys parameter.

Per deprecation timeline; refs #13252.

Note: See TracTickets for help on using tickets.
Back to Top