"ValueError: Lookup failed for model referenced by field app.Model.m2mfield: app.ThroghModel" in simple-self contained M2M setup

[Ramiro Morales]

In a simple ManyToManyField with intermediate model setup involving three models in the same application:

# app_b/models.py
from django.db import models

class Employee(models.Model):
    name = models.CharField(max_length=30)

class Employer(models.Model):
    name = models.CharField(max_length=30)
    employees = models.ManyToManyField(Employee, through='Employment')

class Employment(models.Model):
    employee = models.ForeignKey(Employee)
    employer = models.ForeignKey(Employer)
    recruiting_date = models.DateField()

I'm seeing the following traceback when running migrate after the initial migration has been created for the app:

~/dtest/dtest002$ rm -rf db.sqlite3 app_b/migrations

~/dtest/dtest002$ PYTHONPATH=~/django/upstream python ./manage.py makemigrations
No changes detected

ramiro@serverid:~/dtest/dtest002$ PYTHONPATH=~/django/upstream python ./manage.py makemigrations app_b
Migrations for 'app_b':
  0001_initial.py:
    - Create model Employee
    - Create model Employer
    - Create model Employment

~/dtest/dtest002$ PYTHONPATH=~/django/upstream python ./manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: (none)
  Apply all migrations: admin, contenttypes, app_b, auth, sessions
Synchronizing apps without migrations:
  Creating tables...
  Installing custom SQL...
  Installing indexes...
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying app_b.0001_initial...Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/xxx/django/upstream/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/xxx/django/upstream/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/xxx/django/upstream/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/xxx/django/upstream/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/home/xxx/django/upstream/django/core/management/commands/migrate.py", line 146, in handle
    executor.migrate(targets, plan, fake=options.get("fake", False))
  File "/home/xxx/django/upstream/django/db/migrations/executor.py", line 62, in migrate
    self.apply_migration(migration, fake=fake)
  File "/home/xxx/django/upstream/django/db/migrations/executor.py", line 96, in apply_migration
    migration.apply(project_state, schema_editor)
  File "/home/xxx/django/upstream/django/db/migrations/migration.py", line 107, in apply
    operation.database_forwards(self.app_label, schema_editor, project_state, new_state)
  File "/home/xxx/django/upstream/django/db/migrations/operations/models.py", line 33, in database_forwards
    apps = to_state.render()
  File "/home/xxx/django/upstream/django/db/migrations/state.py", line 89, in render
    model=lookup_model,
ValueError: Lookup failed for model referenced by field app_b.Employer.employees: app_b.Employment

I'm selecting '1.7/beta-2' as Version but I'm running the latest stable/1.7.x code:

~/django/upstream$ git log  -1
commit 30d8b95139a1fa3070f3b112115e624ffcf8e555
Author: Tim Graham <timograham@gmail.com>
Date:   Thu Jun 19 11:02:53 2014 -0400

    [1.7.x] Added feature for implied null (needed for Firebird backend)

    Backport of 1b07781292 from master

Initially Employee didn't exist and the m2m relationship was against a custom User model.

After that (and when trying to create a minimal setup that reproduces the issue):

• First usage of the swappable user model feature was removed and the m2m field was re-pointed to standard auth.User
• And finally to a local Employee model.

In all cases, the same exception was generated.

[Ramiro Morales]

This didn't fail in 1.7b4. git bisect between 1.7b4 and current 1.7 code shows this is a regression introduced in:

commit 31fc34e447137631e6ea58fc33f3642e65479472
Date:   Thu Jun 5 23:03:33 2014 -0700

    [1.7.x] Rewrote migration autodetector to involve actual computer science.
    
    Fixes #22605, #22735; also lays the ground for some other fixes.
    
    Conflicts:
    	django/db/migrations/autodetector.py

Marking as release blocker.

[Andrew Godwin <andrew@…>]

In aa0886e7ae1f372488210e4e51ee6901af8ec643:

[1.7.x] Fixed #22875: Optimizer did not take through= into account.

[Tim Graham <timograham@…>]

In e4c0878b:

Refs #22875 -- Fixed an optimizer test to use a valid scenario.

An explicit intermediary many-to-many relationship must declare forward and
reverse foreign keys. The original issue was in the autodetector as these
operations shouldn't have been generated in this order in the first place
which is tested by AutodetectorTests.test_create_with_through_model.