Add support for non-atomic migrations

[Ludwig Hähne]

Database transactions are always wrapped in transactions (on DBs that support transactional DDL like Postgres). Generally this is a good idea.

However, one can't do batched updates in data migrations which is essential for performing changes on big tables on a live DB:

​http://pankrat.github.io/2015/django-migrations-without-downtimes/

It's also not possible to create indexes concurrently on Postgres from inside a transaction.

Therefore, I'd like to have support for non-atomic migrations in Django because it's pretty messy to work around not having proper support for that:

​http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration

I propose exempting specific migrations from being wrapped in a transaction by setting atomic = False on the migration:

​https://github.com/django/django/compare/master...Ableton:non-atomic-migrations

This was already discussed on the Django developers mailing list:

​https://groups.google.com/forum/#!topic/django-developers/aAYiyAqTlUc

I'll open a pull request for this new feature in a bit.

[Hugo Osvaldo Barrera]

What happens when a non-atomic migrations fails halfway?
Your database is no longer in a state matching this migration, nor the previous one. The database will be left in an undefined, irrecoverable state.

There's no way to roll back from that either.

[Ludwig Hähne]

If the migration doesn't apply without errors, it will not be marked as applied.

It's up to the user to handle those migration errors gracefully (e.g. skip already migrated rows when re-running a previously failed non-atomic migration).

Would it be enough to mention this behaviour in the non-atomic migration documentation?

AFAIU, non-atomic migrations have the same semantics as applying migrations on a database that doesn't support transactional DDL.

[Shai Berger]

#21039 appears to be a special case of this.

[Tim Graham]

Left some comments for improvement on the pull request.

[Ludwig Hähne]

Tim's patch has been applied (thanks!), the tests have been improved and the PR has been rebased.

[Tim Graham <timograham@…>]

In f91a0462:

Fixed #25833 -- Added support for non-atomic migrations.

Added the Migration.atomic attribute which can be set to False
for non-atomic migrations.