makemigrations --update produces migrations with dependencies that don't exist

[Gavin Wahl]

makemigrations --update generates ephemeral migrations that get merged into each app's last migration. If one run produces new migrations in two apps and one depends on the other, the merged migration keeps the dependency on the ephemeral migration's name, which no longer exists after the merge. migrate then fails with NodeNotFoundError. I reproduced on 5.2, 6.1 and master.

1. Two apps with one model each. Run makemigrations.

   Migrations for 'reports':
     reports/migrations/0001_initial.py
       + Create model Report
   Migrations for 'shop':
     shop/migrations/0001_initial.py
       + Create model Product
   

2. Add Region to reports and a foreign key to it on shop.Product:

   class Region(models.Model):
       pass
   

   class Product(models.Model):
       region = models.ForeignKey("reports.Region", models.CASCADE, null=True)
   

3. Run makemigrations --update.

   Migrations for 'reports':
     reports/migrations/0001_initial_updated.py
       + Create model Report
       + Create model Region
   Deleted reports/migrations/0001_initial.py
   Migrations for 'shop':
     shop/migrations/0001_initial_updated.py
       + Create model Product
   Deleted shop/migrations/0001_initial.py
   

4. Run migrate.

   django.db.migrations.exceptions.NodeNotFoundError: Migration shop.0001_initial_updated dependencies reference nonexistent parent node ('reports', '0002_region')
   

shop/0001_initial_updated.py has:

    dependencies = [
        ('reports', '0002_region'),
    ]

reports.0002_region was the ephemeral migration before the merge, never written to disk. The correct dependency is reports.0001_initial_updated. Without --update, the same change produces reports.0002_region and shop.0002_product_region with the right dependency, and migrate works.

It doesn't need a new model on the other side. A new foreign key from shop.Product to the existing reports.Report does the same thing if reports has any change at all in the same run, because makemigrations makes the new shop migration depend on the newest reports migration from that run, and --update merges that one away too.

The cause is in write_to_last_migration_files(). It refuses to update a migration that a migration on disk depends on, but a new migration from the same run is not on disk yet. When new migrations in two apps depend on each other, the dependency is copied onto the merged migration under the ephemeral migration's name, which is never written to disk.

The existing check catches the same dependency once it is on disk. Starting from the two migrations plain makemigrations produces in step 2, a change to reports alone is refused:

CommandError: Cannot update migration 'reports.0002_region' that migrations 'shop.0002_product_region' depend on.

I haven't had too much time to investigate further, but played around with this patch that points dependencies on merged migrations at the migration they were merged into, and refuses an app with more than one new migration. It fixes the steps above on master, and refuses when a cycle would result before deleting anything:

django/core/management/commands/makemigrations.py

@@ -268 +268 @@
         new_changes = {}
         update_previous_migration_paths = {}
         for app_label, app_migrations in changes.items():
+            # More than one new migration in an app means the changes have a
+            # dependency cycle through another app. Merged into one
+            # migration per app the cycle comes back.
+            if len(app_migrations) > 1:
+                raise CommandError(
+                    f"Cannot update last migration of app {app_label}: its changes "
+                    f"need {len(app_migrations)} migrations."
+                )
             # Find last migration.
             leaf_migration_nodes = loader.graph.leaf_nodes(app=app_label)
             if len(leaf_migration_nodes) == 0:
@@ -332 +340 @@
             new_changes[app_label] = [leaf_migration]
             update_previous_migration_paths[app_label] = previous_migration_path
 
+        # A dependency on a migration merged in this run points at the
+        # migration it was merged into.
+        merged = {
+            (app_label, migration.name): (app_label, new_changes[app_label][0].name)
+            for app_label, app_migrations in changes.items()
+            for migration in app_migrations
+        }
+        for [leaf_migration] in new_changes.values():
+            leaf_migration.dependencies = [
+                merged.get(dependency, dependency)
+                for dependency in leaf_migration.dependencies
+            ]
+
         self.write_migration_files(new_changes, update_previous_migration_paths)
 
     def write_migration_files(self, changes, update_previous_migration_paths=None):

I found this with the same fuzzer as #37493 and #37495. It runs makemigrations --update on random model changes across several apps in order to assert that applying migrations generated with --update has the same effect as generating new ones without --update. Any change with new migrations in two apps that depend on each other fails this way.