Opened 7 months ago

Last modified 7 months ago

#37031 assigned Cleanup/optimization

Improve writing migrations guide to adding unique fields on existing table

Reported by: Clifford Gama Owned by: Clifford Gama
Component: Documentation Version: dev
Severity: Normal Keywords: migrations
Cc: MANAS MADESHIYA, Shai Berger, Markus Holtermann Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Clifford Gama)

  1. The current approach splits the work across three files: one to add the field, one to populate values, and one to restore the constraint. All three operations can be placed in a single migration, which is simpler to follow and has the added benefit of being atomic — removing the race condition that is currently warned about in the docs.
  1. I think the section should mention that Field.db_default avoids this problem entirely by having the database generate a unique value per row. This is worth noting upfront so readers can choose the simpler path where their use case allows.
  1. No mention of performant alternatives for large tables: The RunPython example iterates row by row with individual saves. For large tables this will be very slow. The docs should note that QuerySet.bulk_update() or RunSQL are worth considering in that case.

Change History (5)

comment:1 by MANAS MADESHIYA, 7 months ago

Cc: MANAS MADESHIYA added

comment:2 by Sarah Boyce, 7 months ago

Cc: Shai Berger Markus Holtermann added

The docs on "Migrations that add unique fields" was added as part of #23932, I will cc some folks involved in that ticket in case they want to add their opinion

Clifford please can you split this into 2 tickets? Repurpose this ticket only for the "Migrations that add unique fields" suggested changes and a new ticket for "Changing a ManyToManyField to use a through model" suggested changes

comment:3 by Clifford Gama, 7 months ago

Description: modified (diff)
Summary: Improve "Writing migrations" how-to -- unique fields and ManyToManyField through models → Improve writing migrations guide to adding unique fields on existing table
Type: Bug → Cleanup/optimization
Version: 6.0 → dev

comment:4 by Sarah Boyce, 7 months ago

Triage Stage: Unreviewed → Accepted

The basis of these instructions were written ~15 years ago and I agree we can make some improvements. Thank you

comment:5 by Simon Charette, 7 months ago

All three operations can be placed in a single migration, which is simpler to follow and has the added benefit of being atomic — removing the race condition that is currently warned about in the docs.

Some warnings about this approach

  1. This will hold a lock on the table for the whole duration of the transaction which is not practical in many cases. That's the reason why ​we document that data migration should not be run in a transaction.
  2. This won't change anything on MySQL which doesn't support transactional DDL.

I still believe the approach proposed by Shai in #23932:comment:2 is correct.

If you

  1. Add the field as (null=True, unique=True, default=uuid.uuid4) and run the migration
  2. Deploy the changes so any row created from that point sets a value
  3. Run a migration that performs a backfill in a non-atomic matter
  4. Run a migration that drops null=True

There are no race conditions possible.

Version 0, edited 7 months ago by Simon Charette (next)
Note: See TracTickets for help on using tickets.
Back to Top