Opened 9 years ago

Closed 9 years ago

#24794 closed Bug (duplicate)

RunPython parameter "atomic" is ineffective.

Reported by: Vernon Cole Owned by: nobody
Component: Migrations Version: 1.8
Severity: Normal Keywords: RunPython atomic commit
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


This report is similar to #24630, but differs from it in detail. The fix for #24630 is insufficient for this scenario.

I am attempting to apply a data migration to a Postgres table with millions of records -- my first attempt (using RunSQL) failed and rolled back after about two days of run time. It appears that the job must be done in smaller chunks, so I rewrote the migration using RunPython, limiting the size of the result set with a slicing, with set_autocommit(False) and explicit commit calls.

It appears from the documentation that the built-in transaction control can be overridden:

This should be safe, but may cause a crash if you attempt to use the schema_editor provided on these backends; in this case, please set atomic=False.

But "atomic=False" has no effect on the operation of the migration. Stepping through the code using a debugger, I failed to find anyplace where that parameter is actually used, nor could I find a way to short-cut the atomic transaction. It appears that I will need to write a stand-alone program to migrate the data.

Unless someone more talented than me can find a way to make "atomic=False" actually work, it should be removed from the code and from all documentation; or the documentation made more explicit to explain when and where it does work.

Attachments (1) (1.4 KB ) - added by Vernon Cole 9 years ago.
My attempt to break up the migration into smaller chunks.

Download all attachments as: .zip

Change History (3)

by Vernon Cole, 9 years ago

Attachment: added

My attempt to break up the migration into smaller chunks.

comment:1 by Tim Graham, 9 years ago

The atomic parameter doesn't have any effect for databases like PostgreSQL which have features.can_rollback_ddl (see the implementation in 5a917cfef319df33ca30a3b27bd0b0533b8e63bb). I updated my PR in #24630 to make this explicit in the documentation.

comment:2 by Tim Graham, 9 years ago

Resolution: duplicate
Status: newclosed

If you could review my patch and reopen this ticket or leave a comment there if you believe the issue isn't sufficiently addressed, that would be appreciated.

Note: See TracTickets for help on using tickets.
Back to Top