django-admin migrate fails on 2.0 when a project contains squashed migrations from an earlier version

[Dan Collins]

Hello,

Upgrading a project from 1.X to 2.0. One of my apps contains the following migrations:

dcollins@digitalis:~/judgeappstest$ ls exams/migrations/ | grep -v .pyc
0001_squashed_0002_auto_20170519_2125.py
0002_examname.py
0003_exam_editors.py
0004_exam_question_log.py
0005_exam_admin.py
0006_examquestion_topics.py
0007_examtoquestion_option_order.py
0008_exam_for_other_user.py
0009_printed_exams.py
0010_exam_cooldown.py
0011_exam_accepts_suggestions.py
0012_exam_translations.py
0013_examtype_sortkey.py
0014_create_for_others_gk.py
0015_translated_question_reviewed.py
0016_translate_cards.py
0017_card_empty_oracle.py

The first such migration was squashed multiple times (0001 through 000X were squashed, and then that migration was squashed into what we have now). Samples from that migration file:

class Migration(migrations.Migration):
    replaces = [(b'exams', '0001_squashed_0002_auto_20170518_1851'), (b'exams', '0002_auto_20170519_2125')]

         migrations.CreateModel(
             name='ExamQuestion',
             fields=[
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('author', models.ForeignKey(on_delete=models.CASCADE, to=settings.AUTH_USER_MODEL)),
             ],
         ),

After upgrading to Python 3.6/Django 2.0, when attempting to run migrate, I get the following errors:

root@ubuntu-xenial:/dcif# python3 manage.py migrate gatekeeper
System check identified some issues:

WARNINGS:
accounts.DCIProfile.roles: (fields.W340) null has no effect on ManyToManyField.
Operations to perform:
  Apply all migrations: gatekeeper
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 159, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 79, in _create_project_state
    migration.mutate_state(state, preserve=False)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 85, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/fields.py", line 70, in state_forwards
    state.models[app_label, self.model_name_lower].fields.append((self.name, field))
KeyError: ('exams', 'examquestion')

That led me to ​a Stack Overflow answer, and especially kagronick's answer:

I found what causes this and the solution for it. If you have a squashed migration that has a "replaces" property, remove the tuples that reference migrations that don't appear in your django_migrations table. This fixes it.

So, I made one change to the migration file:

diff --git a/exams/migrations/0001_squashed_0002_auto_20170519_2125.py b/exams/migrations/0001_squashed_0002_auto_20170519_2125.py
index 0dd64a8a..1eac171d 100644
--- a/exams/migrations/0001_squashed_0002_auto_20170519_2125.py
+++ b/exams/migrations/0001_squashed_0002_auto_20170519_2125.py
@@ -7,7 +7,7 @@ from django.conf import settings

 class Migration(migrations.Migration):

-    replaces = [(b'exams', '0001_squashed_0002_auto_20170518_1851'), (b'exams', '0002_auto_20170519_2125')]
+    replaces = []

And now the migration is successful:

root@ubuntu-xenial:/dcif# python3 manage.py migrate gatekeeper
System check identified some issues:

WARNINGS:
accounts.DCIProfile.roles: (fields.W340) null has no effect on ManyToManyField.
Operations to perform:
  Apply all migrations: gatekeeper
Running migrations:
  Applying gatekeeper.0007_roleconstraint_invert... OK

I don't know exactly what the cause is here, but I feel like at minimum this shouldn't fail quite so spectacularly. If the issue is really as simple as Django failing to parse a migration file that "replaces" another migration file, which has since been removed from the repository, I would think that would be a bug.

[Tim Graham]

Can you provide a minimal project that reproduces the issue? I'm not sure we can debug the issue without that.

[Dan Collins]

Oh for fuck's sake, it's because 2to3 decided that 'exams' needed to be a bytestring.

replaces = [(b'exams', '0001_squashed_0002_auto_20170518_1851'), (b'exams', '0002_auto_20170519_2125')]

(and of course (b'exams', 'whatever') is not in [('exams', 'whatever'), ...])