Migrations ordering: add constraint run_immediately_after

[Matevž Poljanc]

Problem: after moving models from one app to the other, completely unrelated migrations using RunPython can fail

Details: as our code base is growing, we wanted to refactor the code and split one app (say A) into multiple apps (B and C) (to keep clean separation, reduce interdependency and promote reusability). Because tables were big, we decide to use SeparateDatabaseAndState way (renaming tables in SQL, "deleting" and "creating" models in Python). We have model FirstModel in app A, which we moved to app B using the procedure. This model is also referenced by foreign key on model SecondModel in another app D and have migration which "reconnects" FK from FirstModel app A to FirstModel in app B.

We have the following migrations (in correct dependency order):

1. in app A: SeparateDatabaseAndState to rename table in database from a_firstmodel to b_firstmodel and "delete" FirstModel in app A
2. in app B: SeparateDatabaseAndState to create FirstModel in app B and do nothing in the db
3. in app D: SeparateDatabaseAndState to do nothing in db and "connect" FK field to FirstModel in B

This works well on its own. Now let's add completely unrelated app E, with a migration 0001 using RunPython and no dependencies on any of the other apps. If migration 0001 is run between migrations 2. and 3. it will fail with:
"ValueError: The field d.secondmodel.first_model was declared with a lazy reference to 'a.first_model', but app 'a' doesn't provide model 'first_model'."

I asume this happens because apps.get_model() inside RunPython tries to build the state of all models, which is at that point broken (as migration 3 has not run yet).

There are two fixes, none of which is nice:

1. add dependecies to migration in app D (not nice as we're creating false interdependency, where there truly is none)
2. add run_before to migration 3 and specify migration 0001 in app D (not nice as this could potentially be needed to be updated every time an app is added, otherwise the same problem could arise)

A nice solution could be to add a field such as run_immediatelt_after to migration 3, ensuring migration 3 is always run immediately after 2. This makes sense as those two are intimately connected and no other changes would be needed in code base from that point on.

[Mariusz Felisiak]

Thanks for the report, however SeparateDatabaseAndState is highly specialized operation and you always use it at your own risk. We don't want to add the extra complexity (options) to the Migrations framework to handle niche edge cases.

add dependecies to migration in app D (not nice as we're creating false interdependency, where there truly is none)

Why it's not nice? The migration operations in app D with SeparateDatabaseAndState operations should depend on migrations in app A and B.

If migration 0001 is run between migrations 2. and 3. it will fail with:

I'm not sure how migration 0001 from app E can jump between migration 2 and 3. There must be something wrong with your dependencies (probably manually defined).

Closing as "invalid" (support question) but this could be also interpreted as a new feature in which case I'd close it as "wontfix".

[Simon Charette]

In order to safely relocate a model between apps you have to go through a migration sequence where you have two pointers at the same table (only one managed at a time) and to prevent invalid project states like you are encountering here.

The sequence I've successfully used is the following one

1. In app A, create two migrations (a1, a2). One that renames the model's table and a second one that marks the model unmanaged and deletes it.
2. In app B, create a migrations (b1). Create an unmanaged model that points at the renamed table and then mark it as managed. This migration should depend on a1 and run before a2.
3. In all apps that depend on the relocated model create a migration that uses SeparateDatabaseAndState to repoint FKs. These migrations should depend on b1 and be run before a2.

Unless there are other reasons to add support for run_immediatelt_after (which would complexity our dependency resolving logic) I would consider this ticket more a duplicate of #24686 which tracks adding support for moving models between apps.

One note I would add is that if were were able to make the logic backing AlterField smart enough to be a noop when it's repointed to a different model backed by the same table we should be able to achieve model relocation without the use of SeparateDatabaseAndState which was contention point of the implementation proposed for #24686.