Squashed migrations not used always when testing

[anonymous]

I've squashed migrations in my apps, did some fixing and to test them on clean database I was just running tests. When having old and squashes in migrations folders some part of django-admin test ... resulted in KeyErrors like the one below. Some did pass and used the squashed migrations.

Traceback (most recent call last):
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/coverage/execfile.py", line 104, in run_python_file
    exec_code_object(code, main_mod.__dict__)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/coverage/backward.py", line 93, in exec_code_object
    exec(code, global_map)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/commands/test.py", line 50, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/commands/test.py", line 71, in execute
    super(Command, self).execute(*args, **options)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django_jenkins/management/commands/jenkins.py", line 98, in handle
    failures = test_runner.run_tests(test_labels)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/test/runner.py", line 147, in run_tests
    old_config = self.setup_databases()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django_jenkins/runner.py", line 127, in setup_databases
    return super(CITestSuiteRunner, self).setup_databases()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/test/runner.py", line 109, in setup_databases
    return setup_databases(self.verbosity, self.interactive, **kwargs)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/test/runner.py", line 299, in setup_databases
    serialize=connection.settings_dict.get("TEST_SERIALIZE", True),
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/backends/creation.py", line 374, in create_test_db
    test_flush=True,
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/__init__.py", line 115, in call_command
    return klass.execute(*args, **defaults)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/commands/migrate.py", line 63, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/migrations/executor.py", line 16, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/migrations/loader.py", line 48, in __init__
    self.build_graph()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/migrations/loader.py", line 216, in build_graph
    normal[child_key].dependencies.remove(replaced)
KeyError: ('buku_notification', '0003_auto_20140424_1540')

Where buku_notification.0003_auto_20140424_1540 is one of migrations that got squashed, it listed in "replaces" of its squash migration. I can't reproduce it with my second commit that deleted all migrations that got squashed.

[Tim Graham]

More details like a minimal project that reproduces the error would be really helpful in debugging the issue.

[Piotr Maliński]

Will try to provide.

[Piotr Maliński]

I've traced the problem a bit and it looks like it's not the order of migrations to be the source of the problem.

Step to reproduce:

• squash few migrations in an application and apply all migrations needed
• remove squashed migrations
• make a new migration and apply it
• squash (old squash + new migration)
• (you may remove squashed migrations too)
• look at the replaces list of that squash and try to run test so that it will be applied to a clean db

squashmigrations will copy replaces contents from squashed migrations. Migrations listed there may do not exist and migrating will result in key errors (although sometimes not). The squashmigrations command and documentation should probably mention that replaces should be removed when squashed migrations are removed from the application

[Tim Graham]

Andrew, can you confirm the desired behavior?

[Andrew Godwin]

Yes, riklaunim is correct that you're meant to remove replaces from a squashed migration before re-squashing (you can't squash a squashed replacement migration, so you have to fully transition it to a normal migration first).

It's just a doc fix. I'll do it.

[Simon Charette]

I don't know if it's already the case but would it be possible to prevent the migration framework from squashing replacement migrations by looking up if they have a replaces attribute?

[Andrew Godwin]

That is, indeed, exactly what I am implementing right now.

[Andrew Godwin <andrew@…>]

In 8ebd6d35d77c1c535a9ffe8151445ad853845b61:

Fixed #23090: Document and enforce not double-squashing migrations

[Andrew Godwin <andrew@…>]

In 3deddc2fdf1f47a9564b6d39ee53dc5bdd944dc2:

[1.7.x] Fixed #23090: Document and enforce not double-squashing migrations