"ProgrammingError: relation does not exist" when renaming many-to-many target model

[Danilo Bargen]

To demonstrate this bug, I created a Github repository: ​https://github.com/dbrgn/django18-migration-bug You can see each step in the commit log.

When I run the migrations for that project in a clean Postgres database, I get an exception. You can run "reproduce.sh" to reproduce (d'oh!).

$ dropdb dj18
$ createdb dj18
$ ./manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: staticfiles, messages
  Apply all migrations: admin, debug, contenttypes, auth, sessions
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying debug.0001_initial... OK
  Applying debug.0002_auto_20150429_1317...Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/core/management/base.py", line 390, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute
    output = self.handle(*args, **options)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 221, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/migrations/executor.py", line 147, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/migrations/operations/models.py", line 217, in database_forwards
    to_field,
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 469, in alter_field
    return self._alter_many_to_many(model, old_field, new_field, strict)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 780, in _alter_many_to_many
    new_field.rel.through._meta.get_field(new_field.m2m_reverse_field_name()),
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 484, in alter_field
    old_db_params, new_db_params, strict)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 715, in _alter_field
    self.execute(self._create_fk_sql(model, new_field, "_fk_%(to_table)s_%(to_column)s"))
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 107, in execute
    cursor.execute(sql, params)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/danilo/.virtualenvs/django18-migration-bug/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "debug_spam" does not exist

This only happens in Django 1.8, with 1.7.7 the migrations run fine.

[Danilo Bargen]

Note that both migrations were auto-generated:

$ ./manage.py makemigrations debug
Did you rename the debug.Spam model to Bacon? [y/N] y
Migrations for 'debug':
  0002_auto_20150429_1331.py:
    - Rename model Spam to Bacon

[Marten Kenbeek]

On sqlite the initial migration works, but subsequent calls to migrate or makemigrations produce this traceback.

[Tim Graham]

At 1aa3e09c2043c88a760e8b73fb95dc8f1ffef50e, the attached test for Django's test suite starts failing with "AttributeError: 'str' object has no attribute '_meta'", then the "relation does not exist" traceback starts happening at fb48eb05816b1ac87d58696cdfe48be18c901f16.

[Markus Holtermann]

I localized the error to these four lines: ​https://github.com/django/django/blob/master/django/db/migrations/operations/models.py#L162-L165

They wrongly filter out <ManyToManyRel: debug.john> on the model debug.Spam (i.e. __fake__.Spam):

ipdb> model._meta.get_fields(include_hidden=True)
(<ManyToOneRel: debug.john_consumed_food>, <ManyToManyRel: debug.john>, <django.db.models.fields.AutoField: id>, <django.db.models.fields.CharField: description>)

ipdb> field = model._meta.get_fields(include_hidden=True)[0]
ipdb> field
<ManyToOneRel: debug.john_consumed_food>
ipdb> field.auto_created
True
ipdb> field.concrete
False
ipdb> field.hidden
True
ipdb> field.many_to_many
False

ipdb> field = model._meta.get_fields(include_hidden=True)[1]
ipdb> field
<ManyToManyRel: debug.john>
ipdb> field.auto_created
True
ipdb> field.concrete
False
ipdb> field.hidden
False
ipdb> field.many_to_many
True

[Markus Holtermann]

​https://github.com/django/django/pull/4584

[Markus Holtermann <info@…>]

In 63f9b633:

Fixed #24725 -- Allowed renaming of target models in ManyToMany relations

This is a regression caused by introducing rendered migration states in
1aa3e09c2043 and the _meta refactoring in fb48eb05816b.

Thanks to Danilo Bargen for reporting the issue and Marten Kenbeek and
Tim Graham for triaging the bug and providing the initial test case.

[Danilo Bargen]

Brilliant, thanks for the great work and the quick reaction :)

Any idea when a 1.8.1 release can be expected?

[Markus Holtermann <info@…>]

In bf4dc573:

[1.8.x] Fixed #24725 -- Allowed renaming of target models in ManyToMany relations

This is a regression caused by introducing rendered migration states in
1aa3e09c2043 and the _meta refactoring in fb48eb05816b.

Thanks to Danilo Bargen for reporting the issue and Marten Kenbeek and
Tim Graham for triaging the bug and providing the initial test case.

Backport of 63f9b633f9cb620a4c4764b47dca127706a8d7b4 from master

[Markus Holtermann]

Replying to dbrgn:

Brilliant, thanks for the great work and the quick reaction :)

Any idea when a 1.8.1 release can be expected?

You're welcome. If nothing unexpected happens 1.8.1 is scheduled for tomorrow, iirc.