Foreignkey does (sometimes) not get added to migration of M2M with a through model

[Vidir Valberg Gudmundsson]

The following models:

from django.conf import settings
from django.db import models


class Event(models.Model):

    participants = models.ManyToManyField(
        settings.AUTH_USER_MODEL,
        through='testapp.Participant',
        related_name='events'
    )


class Participant(models.Model):

    event = models.ForeignKey(
        'testapp.Event',
        related_name='associations',
    )

    user = models.ForeignKey(
        settings.AUTH_USER_MODEL
    )

Result in the following migration

$ ./manage.py makemigrations           
Migrations for 'testapp':
  0001_initial.py:
    - Create model Event
    - Create model Participant

$ cat testapp/migrations/0001_initial.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations
from django.conf import settings


class Migration(migrations.Migration):

    dependencies = [
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
    ]

    operations = [
        migrations.CreateModel(
            name='Event',
            fields=[
                ('id', models.AutoField(serialize=False, auto_created=True, primary_key=True, verbose_name='ID')),
                ('participants', models.ManyToManyField(through='testapp.Participant', to=settings.AUTH_USER_MODEL)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Participant',
            fields=[
                ('id', models.AutoField(serialize=False, auto_created=True, primary_key=True, verbose_name='ID')),
                ('event', models.ForeignKey(to_field='id', to='testapp.Event')),
                ('user', models.ForeignKey(to_field='id', to=settings.AUTH_USER_MODEL)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
    ]

When migrating the following error occurs:

$ ./manage.py migrate       
Operations to perform:
  Synchronize unmigrated apps: sessions, auth, admin, contenttypes
  Apply all migrations: testapp
Synchronizing apps without migrations:
  Creating tables...
    Creating table django_admin_log
    Creating table auth_permission
    Creating table auth_group_permissions
    Creating table auth_group
    Creating table auth_user_groups
    Creating table auth_user_user_permissions
    Creating table auth_user
    Creating table django_content_type
    Creating table django_session
  Installing custom SQL...
  Installing indexes...
Running migrations:
  Applying testapp.0001_initial...Traceback (most recent call last):
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/apps/config.py", line 152, in get_model
    return self.models[model_name.lower()]
KeyError: 'participant'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/db/migrations/state.py", line 76, in render
    model = self.apps.get_model(lookup_model[0], lookup_model[1])
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/apps/registry.py", line 190, in get_model
    return self.get_app_config(app_label).get_model(model_name.lower())
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/apps/config.py", line 155, in get_model
    "App '%s' doesn't have a '%s' model." % (self.label, model_name))
LookupError: App 'testapp' doesn't have a 'participant' model.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/core/management/commands/migrate.py", line 146, in handle
    executor.migrate(targets, plan, fake=options.get("fake", False))
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/db/migrations/executor.py", line 62, in migrate
    self.apply_migration(migration, fake=fake)
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/db/migrations/executor.py", line 96, in apply_migration
    migration.apply(project_state, schema_editor)
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/db/migrations/migration.py", line 107, in apply
    operation.database_forwards(self.app_label, schema_editor, project_state, new_state)
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/db/migrations/operations/models.py", line 33, in database_forwards
    apps = to_state.render()
  File "/home/valberg/.virtualenvs/django/lib/python3.3/site-packages/django/db/migrations/state.py", line 86, in render
    model=lookup_model,
ValueError: Lookup failed for model referenced by field testapp.Event.participants: testapp.Participant

The odd thing is that the following models (same structure as above, just different names):

from django.conf import settings
from django.db import models


# Same as Event above
class Foo(models.Model):

    users = models.ManyToManyField(
        settings.AUTH_USER_MODEL,
        through='testapp.Bar',
        related_name='foos'
    )


# Same as Participant above
class Bar(models.Model):

    foo = models.ForeignKey(
        'testapp.Foo',
        related_name='bars',
    )

    user = models.ForeignKey(
        settings.AUTH_USER_MODEL
    )

Result in the following:

$ ./manage.py makemigrations           
Migrations for 'testapp':
  0001_initial.py:
    - Create model Bar
    - Create model Foo
    - Add field foo to bar

$ cat testapp/migrations/0001_initial.py 
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations
from django.conf import settings


class Migration(migrations.Migration):

    dependencies = [
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
    ]

    operations = [
        migrations.CreateModel(
            name='Bar',
            fields=[
                ('id', models.AutoField(serialize=False, primary_key=True, auto_created=True, verbose_name='ID')),
                ('user', models.ForeignKey(to_field='id', to=settings.AUTH_USER_MODEL)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Foo',
            fields=[
                ('id', models.AutoField(serialize=False, primary_key=True, auto_created=True, verbose_name='ID')),
                ('users', models.ManyToManyField(to=settings.AUTH_USER_MODEL, through='testapp.Bar')),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.AddField(
            model_name='bar',
            name='foo',
            field=models.ForeignKey(to_field='id', to='testapp.Foo'),
            preserve_default=True,
        ),
    ]

$ ./manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: admin, auth, sessions, contenttypes
  Apply all migrations: testapp
Synchronizing apps without migrations:
  Creating tables...
    Creating table django_admin_log
    Creating table auth_permission
    Creating table auth_group_permissions
    Creating table auth_group
    Creating table auth_user_groups
    Creating table auth_user_user_permissions
    Creating table auth_user
    Creating table django_content_type
    Creating table django_session
  Installing custom SQL...
  Installing indexes...
Running migrations:
  Applying testapp.0001_initial... OK

So somehow the naming of the models has a say in how the migrations are made. In the failing example there is no creation of the FK from Participant to Event as there is from Bar to Foo.

[Baptiste Mispelon]

Hi,

I can reproduce your issue with this simplified models (no reference to settings.AUTH_USER_MODEL):

from django.db import models

class Foo(models.Model):
    pass


class Bar(models.Model):
    foos = models.ManyToManyField('Foo', through='FooBar')


class FooBar(models.Model):
    foo = models.ForeignKey('Foo')
    bar = models.ForeignKey('Bar')

As you mentionned, renaming FooBar to AFooBar (so that it appears before when sorting alphabetically) makes everything work.

I'll upgrade the ticket to a release blocker.

Thanks.

[Andrew Godwin]

Looks like there's not enough dependency stuff set in the autodetector to force the operation ordering. Hopefully an easy-ish fix.

[Andrew Godwin <andrew@…>]

In 3f91238adf819b1d86f800d6b535b3c04447dac2:

Fixed #22777: Add dependency on through for autodetected M2M adds

[Andrew Godwin <andrew@…>]

In 55fa4c2d3423b051fce5663b3efc13e458fbb777:

[1.7.x] Fixed #22777: Add dependency on through for autodetected M2M adds