Opened 3 years ago

Closed 3 years ago

#25833 closed New feature (fixed)

Add support for non-atomic migrations

Reported by: Ludwig Hähne Owned by: nobody
Component: Migrations Version: master
Severity: Normal Keywords:
Cc: Shai Berger Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Ludwig Hähne)

Database transactions are always wrapped in transactions (on DBs that support transactional DDL like Postgres). Generally this is a good idea.

However, one can't do batched updates in data migrations which is essential for performing changes on big tables on a live DB:

http://pankrat.github.io/2015/django-migrations-without-downtimes/

It's also not possible to create indexes concurrently on Postgres from inside a transaction.

Therefore, I'd like to have support for non-atomic migrations in Django because it's pretty messy to work around not having proper support for that:

http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration

I propose exempting specific migrations from being wrapped in a transaction by setting atomic = False on the migration:

https://github.com/django/django/compare/master...Ableton:non-atomic-migrations

This was already discussed on the Django developers mailing list:

https://groups.google.com/forum/#!topic/django-developers/aAYiyAqTlUc

I'll open a pull request for this new feature in a bit.

Change History (8)

comment:1 Changed 3 years ago by Ludwig Hähne

Description: modified (diff)

comment:2 Changed 3 years ago by Simon Charette

Triage Stage: UnreviewedAccepted

comment:3 Changed 3 years ago by Hugo Osvaldo Barrera

What happens when a non-atomic migrations fails halfway?
Your database is no longer in a state matching this migration, nor the previous one. The database will be left in an undefined, irrecoverable state.

There's no way to roll back from that either.

comment:4 Changed 3 years ago by Ludwig Hähne

If the migration doesn't apply without errors, it will not be marked as applied.

It's up to the user to handle those migration errors gracefully (e.g. skip already migrated rows when re-running a previously failed non-atomic migration).

Would it be enough to mention this behaviour in the non-atomic migration documentation?

AFAIU, non-atomic migrations have the same semantics as applying migrations on a database that doesn't support transactional DDL.

comment:5 Changed 3 years ago by Shai Berger

Cc: Shai Berger added

#21039 appears to be a special case of this.

comment:6 Changed 3 years ago by Tim Graham

Patch needs improvement: set
Summary: Non-atomic migrationsAdd support for non-atomic migrations

Left some comments for improvement on the pull request.

comment:7 Changed 3 years ago by Ludwig Hähne

Patch needs improvement: unset

Tim's patch has been applied (thanks!), the tests have been improved and the PR has been rebased.

comment:8 Changed 3 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In f91a0462:

Fixed #25833 -- Added support for non-atomic migrations.

Added the Migration.atomic attribute which can be set to False
for non-atomic migrations.

Note: See TracTickets for help on using tickets.
Back to Top