Opened 6 years ago

Closed 5 years ago

Last modified 3 years ago

#29721 closed Bug (fixed)

Migrations are not applied atomically

Reported by: Gavin Wahl Owned by: Sanyam Khurana
Component: Migrations Version: 2.1
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Migrations run the migration and record that the migration was applied in a different transaction. This results in a broken/inconsistent database when an error or crash happens between those two steps. This is not purely theoretical, it has happened to me.

The executor needs to call migration.apply and record_applied in the same transaction, and then the django_migrations table can't get out of sync with the migrations that have actually been applied. I think this could be done by moving the call to record_applied inside the schema_editor context manager block.

Change History (11)

comment:1 by Simon Charette, 6 years ago

Triage Stage: Unreviewed → Accepted

I guess we could move the recording logic in the context manager to be entirely correct but I'm surprised this has happened to you given the small amount of operations performed between the two blocks.

Are you certain you didn't hit ​the documented caveat of MySQL regarding atomic migrations? In that case this change wouldn't help at all. Was it the django_migrations table creation that failed?

Are you able to provide a patch/PR?

Last edited 6 years ago by Simon Charette (previous) (diff)

comment:2 by Gavin Wahl, 6 years ago

It was on postgres. It was a regular migration adding some tables, and the tables were added but the migration was not recorded in django_migrations.

comment:3 by Simon Charette, 6 years ago

This is really strange, are you sure the migration was not marked as atomic=False?

If not It's unlikely to have happened without an exception being surfaced.

comment:4 by Tim Graham, 6 years ago

Type: Uncategorized → Bug

comment:5 by Sanyam Khurana, 6 years ago

Owner: changed from nobody to Sanyam Khurana
Status: new → assigned

I'm working on this during DjangoCon US sprints and would submit a patch ASAP.

comment:6 by Simon Charette, 6 years ago

Has patch: set
Triage Stage: Accepted → Ready for checkin

​PR. Given the difficulty of adding a test that isn't highly implementation specific and how the new implementation is technically more correct I consider this patch RFC.

Last edited 5 years ago by Tim Graham (previous) (diff)

comment:7 by Simon Charette, 6 years ago

Needs tests: set
Triage Stage: Ready for checkin → Accepted

Removing the RFC status as a test for backends supporting transactional DDL can probably be added.

comment:8 by Simon Charette, 5 years ago

Needs tests: unset
Triage Stage: Accepted → Ready for checkin

comment:9 by Tim Graham <timograham@…>, 5 years ago

Resolution: → fixed
Status: assigned → closed

In c86a3d80:

Fixed #29721 -- Ensured migrations are applied and recorded atomically.

comment:10 by Mariusz Felisiak <felisiak.mariusz@…>, 3 years ago

In 533a583:

Refs #29721 -- Simplified migration used to test atomic recording.

This makes sure atomic recording of migration application is used when
the schema editor doesn't defer any statement.

comment:11 by Mariusz Felisiak <felisiak.mariusz@…>, 3 years ago

In 900b2ce9:

[3.2.x] Refs #29721 -- Simplified migration used to test atomic recording.

This makes sure atomic recording of migration application is used when
the schema editor doesn't defer any statement.

Backport of 533a5835784b95335c8373b6d0b9495b3834e96e from master

Note: See TracTickets for help on using tickets.
Back to Top