Opened 2 years ago

Closed 2 years ago

#25930 closed Bug (fixed)

Django-migrations & sqlite3 fail when migration-history is non-linear

Reported by: Klaas van Schelven Owned by: nobody
Component: Migrations Version:
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

  1. We started seeing problems in our tests (which were the only place where we used sqlite3), when running the tests with all migrations (as is the default). The problem as it showed up: lots of tests failing, complaining that a certain column wasn't existing in the DB. Despite the fact that a migration existed to create the column, and despite the fact that the migrations seemed fine.
  1. As [stated in the documentation](https://docs.djangoproject.com/en/1.9/topics/migrations/#sqlite), Django doesnt' actually run alter table statements when altering sqlite tables via migrations. Rather, it fully recreates the tables based on it
  1. Consider the following graph (oldest migration at the bottom):
    M
    | 
   / \
  B   C
   \ /
    |
    A

For example's sake, let's assume B and C add columns to some table b and c respectively. M does nothing, it's only there to provide a single sink/target for django-migrations. I.e. it's a migration which only consists of dependencies. Let's assume that Django-migrations chooses the ordering A -> B -> C -> M (this is non-essential, the analogous problem will show up with the other possible ordering (C before B).

  1. As far as I understand, django-migrations keeps track of an internal representation of the table after each migration; after migration B this contains b but not c; after migration C this contains c but not b; after migration M this contains both b and c.
  1. Migrations B and C contain alterations, triggering a full recreation of the table. (This is the behavior as stated in the documentation, and I've also seen this in practice by printing all generated queries)
  1. At migration C this means: a full recreation of the table, but without the column b. This is the source of problem (the problem being: the complaints about column b not existing)
  1. Migration M contains nothing, triggering nothing, even though django-migrations has the right view of what the table should look like happen at this point. (it has the merged view of both paths). This is the point in time where the problem isn't solved, even though it could be.
  1. Another look at point '7' reveals that it can be used for a workaround: add a statement that amounts to "alter to the status quo", i.e. a statement that looks like an alter statement but doesn't actually alter anything. This statement will force a full recreation of the table, containing both columns b and c. Even though I haven't deeply inpected how django-migrations works, the workaround works. This causes me to think that the hypothesis is indeed correct.

I've observed the above on Django 1.7; I have not checked other installations for similar problems.

Observed in the wild, including the workaround; I do not currently have the time available to provide an executable clean test which does not refer to our production-code.

Still, I hope the write-up of the problem as I understand it is useful

Change History (4)

comment:1 Changed 2 years ago by Simon Charette

Version: 1.8

Hi @vanschelven,

Thanks to your very detailed report I managed to reproduce against the latest Django 1.7.

The good news is that while I didn't bisect the exact commit the issue seems to be fixed on Django 1.8 which is the oldest supported version at this point.

Please confirm 1.8 solves your reported issue. If it does please resolve this ticket as invalid as Django 1.7 is not supported anymore.

Last edited 2 years ago by Simon Charette (previous) (diff)

comment:2 Changed 2 years ago by Simon Charette

Component: UncategorizedMigrations
Type: UncategorizedBug

comment:3 Changed 2 years ago by Baptiste Mispelon

Seems to have been fixed by 1aa3e09c2043c88a760e8b73fb95dc8f1ffef50e (found using Simon's reproduction steps).

comment:4 Changed 2 years ago by Tim Graham

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top