Add an --ignore-deps flag to squashmigrations

[Shai Berger]

In large, long-going projects -- the ones whose need for migration squashing is the greatest -- apps have a tendency to develop dependencies upon each other. This entails problems with migrations -- dependencies could be circular, or an app could be removed from the project but the migrations that hold dependencies on it still remain, and squashmigrations faces constraints that makes it very hard to do correctly. Current implementation often generates migrations that make no sense, with dependencies on several migrations from the same app.

As a partial solution, let's add a --ignore-deps flag to squashmigrations. When this flag is applied, the generated squashed migration will

• Only depend on the dependencies of the first migration in the range squashed
• Not include the operations that create or modify relation fields, if these fields refer to models from other apps (deletion can be included)

The idea is that such a migration will be correct, but not complete. Assuming the range of migrations includes the last migrations of the app, it will then be possible to add the missing pieces by a regular makemigrations; but the squashing will still include all the non-ellidable RunSQL and RunPython operations -- the squashmigrations --ignore-deps + makemigrations dance is proposed as a safer alternative to "migration bankruptcy", the practice of just removing all the migrations and stating afresh

On a side note: It may be safer to add an operation which enforces that it is indeed called with the last migration of the app in range, and maybe even automatically invokes the following makemigrations, but I think we should trust developers with the lower-level building bricks, which may be useful in other situations as well.

This follows from a ​discussion on the forum.

[Clinton Christian]

I'm working on this and will finalize a PR shortly.

[Sarah Boyce]

Hi Shai, thank you for raising this ticket
cc-ing a few people here in case they have further insights or opinions on this
Tentatively accepting as the discussion seems positive and I can see the value of the feature 👍

[Clinton Christian]

Filtering out multi-app FieldOperations is pretty straight forward, but should we mutate a ModelOperation to exclude multi-app foreign key references, or exclude the ModelOperation entirely?

[Shai Berger]

Replying to Clinton Christian:

Filtering out multi-app FieldOperations is pretty straight forward, but should we mutate a ModelOperation to exclude multi-app foreign key references, or exclude the ModelOperation entirely?

Coming back to this after two years, with the response I put on Clinton's ​PR:

Many ModelOperations are, in fact, irrelevant, because they do not operate on fields; so it is better to focus at the more-specific operation level. and even then, it changes by case. In the case of CreateModel, it is better to modify the operation, because it is likely that migrations down the line will rely on the model existing -- even if they don't rely on the relation field. In the case of Index or Constraint operations (some of which, by the way, do not inherit ModelOperation), it is better to remove the whole operation, because the constraint or index will be entirely incorrect if it is missing a field.

[Shai Berger]

Thinking more about that deferral-of-relations issue:

I think the operations and parameters that are filtered out should be recorded, as comments, in the migration file. I suspect in many cases, later operations -- e.g. {{{RunPython}} -- will rely on the results of the deferred operations, and if there's no record of their removal in the migration file, the errors will be quite difficult to debug.

This requires some support in the migration-writing machinery.

[Shai Berger]

And one more related, but separate idea:

It may be better to optimize the squashed migration before filtering it for external dependencies -- if a dependency is added and then removed, it is better to have it "organically" deleted as an optimization, rather than have just the addition removed as a "no-deps" effect.