Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#23090 closed Bug (fixed)

Squashed migrations not used always when testing

Reported by: anonymous Owned by: Andrew Godwin
Component: Migrations Version: 1.7-rc-1
Severity: Release blocker Keywords:
Cc: Simon Charette Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I've squashed migrations in my apps, did some fixing and to test them on clean database I was just running tests. When having old and squashes in migrations folders some part of django-admin test ... resulted in KeyErrors like the one below. Some did pass and used the squashed migrations.

Traceback (most recent call last):
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/coverage/execfile.py", line 104, in run_python_file
    exec_code_object(code, main_mod.__dict__)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/coverage/backward.py", line 93, in exec_code_object
    exec(code, global_map)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/commands/test.py", line 50, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/commands/test.py", line 71, in execute
    super(Command, self).execute(*args, **options)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django_jenkins/management/commands/jenkins.py", line 98, in handle
    failures = test_runner.run_tests(test_labels)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/test/runner.py", line 147, in run_tests
    old_config = self.setup_databases()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django_jenkins/runner.py", line 127, in setup_databases
    return super(CITestSuiteRunner, self).setup_databases()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/test/runner.py", line 109, in setup_databases
    return setup_databases(self.verbosity, self.interactive, **kwargs)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/test/runner.py", line 299, in setup_databases
    serialize=connection.settings_dict.get("TEST_SERIALIZE", True),
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/backends/creation.py", line 374, in create_test_db
    test_flush=True,
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/__init__.py", line 115, in call_command
    return klass.execute(*args, **defaults)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/core/management/commands/migrate.py", line 63, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/migrations/executor.py", line 16, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/migrations/loader.py", line 48, in __init__
    self.build_graph()
  File "/tmp/buku-test-virtualenv-908c498ca6f66e231f0be70fcb6b51b2-5/lib/python3.3/site-packages/django/db/migrations/loader.py", line 216, in build_graph
    normal[child_key].dependencies.remove(replaced)
KeyError: ('buku_notification', '0003_auto_20140424_1540')

Where buku_notification.0003_auto_20140424_1540 is one of migrations that got squashed, it listed in "replaces" of its squash migration. I can't reproduce it with my second commit that deleted all migrations that got squashed.

Change History (10)

comment:1 Changed 4 years ago by Piotr Maliński

comment:2 Changed 4 years ago by Tim Graham

More details like a minimal project that reproduces the error would be really helpful in debugging the issue.

comment:3 Changed 4 years ago by Piotr Maliński

Will try to provide.

comment:4 Changed 4 years ago by Piotr Maliński

I've traced the problem a bit and it looks like it's not the order of migrations to be the source of the problem.

Step to reproduce:

  • squash few migrations in an application and apply all migrations needed
  • remove squashed migrations
  • make a new migration and apply it
  • squash (old squash + new migration)
  • (you may remove squashed migrations too)
  • look at the replaces list of that squash and try to run test so that it will be applied to a clean db

squashmigrations will copy replaces contents from squashed migrations. Migrations listed there may do not exist and migrating will result in key errors (although sometimes not). The squashmigrations command and documentation should probably mention that replaces should be removed when squashed migrations are removed from the application

comment:5 Changed 4 years ago by Tim Graham

Severity: NormalRelease blocker
Triage Stage: UnreviewedAccepted
Type: UncategorizedBug

Andrew, can you confirm the desired behavior?

comment:6 Changed 4 years ago by Andrew Godwin

Owner: changed from nobody to Andrew Godwin
Status: newassigned

Yes, riklaunim is correct that you're meant to remove replaces from a squashed migration before re-squashing (you can't squash a squashed replacement migration, so you have to fully transition it to a normal migration first).

It's just a doc fix. I'll do it.

comment:7 Changed 4 years ago by Simon Charette

Cc: Simon Charette added

I don't know if it's already the case but would it be possible to prevent the migration framework from squashing replacement migrations by looking up if they have a replaces attribute?

comment:8 Changed 4 years ago by Andrew Godwin

That is, indeed, exactly what I am implementing right now.

comment:9 Changed 4 years ago by Andrew Godwin <andrew@…>

Resolution: fixed
Status: assignedclosed

In 8ebd6d35d77c1c535a9ffe8151445ad853845b61:

Fixed #23090: Document and enforce not double-squashing migrations

comment:10 Changed 4 years ago by Andrew Godwin <andrew@…>

In 3deddc2fdf1f47a9564b6d39ee53dc5bdd944dc2:

[1.7.x] Fixed #23090: Document and enforce not double-squashing migrations

Note: See TracTickets for help on using tickets.
Back to Top