Django-migrations & sqlite3 fail when migration-history is non-linear

[Klaas van Schelven]

1. We started seeing problems in our tests (which were the only place where we used sqlite3), when running the tests with all migrations (as is the default). The problem as it showed up: lots of tests failing, complaining that a certain column wasn't existing in the DB. Despite the fact that a migration existed to create the column, and despite the fact that the migrations seemed fine.

2. As [stated in the documentation](​https://docs.djangoproject.com/en/1.9/topics/migrations/#sqlite), Django doesnt' actually run alter table statements when altering sqlite tables via migrations. Rather, it fully recreates the tables based on it

3. Consider the following graph (oldest migration at the bottom):

    M
    | 
   / \
  B   C
   \ /
    |
    A

For example's sake, let's assume B and C add columns to some table b and c respectively. M does nothing, it's only there to provide a single sink/target for django-migrations. I.e. it's a migration which only consists of dependencies. Let's assume that Django-migrations chooses the ordering A -> B -> C -> M (this is non-essential, the analogous problem will show up with the other possible ordering (C before B).

4. As far as I understand, django-migrations keeps track of an internal representation of the table after each migration; after migration B this contains b but not c; after migration C this contains c but not b; after migration M this contains both b and c.

5. Migrations B and C contain alterations, triggering a full recreation of the table. (This is the behavior as stated in the documentation, and I've also seen this in practice by printing all generated queries)

6. At migration C this means: a full recreation of the table, but without the column b. This is the source of problem (the problem being: the complaints about column b not existing)

7. Migration M contains nothing, triggering nothing, even though django-migrations has the right view of what the table should look like happen at this point. (it has the merged view of both paths). This is the point in time where the problem isn't solved, even though it could be.

8. Another look at point '7' reveals that it can be used for a workaround: add a statement that amounts to "alter to the status quo", i.e. a statement that looks like an alter statement but doesn't actually alter anything. This statement will force a full recreation of the table, containing both columns b and c. Even though I haven't deeply inpected how django-migrations works, the workaround works. This causes me to think that the hypothesis is indeed correct.

I've observed the above on Django 1.7; I have not checked other installations for similar problems.

Observed in the wild, including the workaround; I do not currently have the time available to provide an executable clean test which does not refer to our production-code.

Still, I hope the write-up of the problem as I understand it is useful

[Simon Charette]

Hi @vanschelven,

Thanks to your very detailed report I managed ​to reproduce against the latest Django 1.7.

The good news is that while I didn't bisect the exact commit but the issue seems to be fixed on Django 1.8 which is the oldest supported version at this point.

Please confirm 1.8 solves your reported issue. If it does please resolve this ticket as invalid as Django 1.7 is not supported anymore.

[Baptiste Mispelon]

Seems to have been fixed by 1aa3e09c2043c88a760e8b73fb95dc8f1ffef50e (found using Simon's reproduction steps).