Django 1.9 regression in migrations due to lazy model operations refactor

[Brian May]

With open source project called spud.

Generates Exception:

FieldDoesNotExist: album has no field named u'album_id'

This field was removed. However in the migration that is being run when it fails, it did exist. So as a result, I suspect this isn't #26180.

The migration in question attempts to rename the album_id field to id.

Works fine with Django < =1.8

Stack Trace:

/usr/lib/python2.7/dist-packages/pytest_django/fixtures.py:54: in _django_db_setup
    interactive=False)
/tmp/deleteme/local/lib/python2.7/site-packages/django/test/runner.py:726: in setup_databases
    serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/backends/base/creation.py:70: in create_test_db
    run_syncdb=True,
/tmp/deleteme/local/lib/python2.7/site-packages/django/core/management/__init__.py:119: in call_command
    return command.execute(*args, **defaults)
/tmp/deleteme/local/lib/python2.7/site-packages/django/core/management/base.py:399: in execute
    output = self.handle(*args, **options)
/tmp/deleteme/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py:200: in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/migrations/executor.py:92: in migrate
    self._migrate_all_forwards(plan, full_plan, fake=fake, fake_initial=fake_initial)
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/migrations/executor.py:121: in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/migrations/executor.py:198: in apply_migration
    state = migration.apply(state, schema_editor)
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/migrations/migration.py:123: in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/migrations/operations/fields.py:275: in database_forwards
    to_model._meta.get_field(self.new_name),
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/backends/base/schema.py:482: in alter_field
    old_db_params, new_db_params, strict)
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/backends/sqlite3/schema.py:245: in _alter_field
    self._remake_table(model, alter_fields=[(old_field, new_field)])
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/backends/sqlite3/schema.py:181: in _remake_table
    self.create_model(temp_model)
/tmp/deleteme/local/lib/python2.7/site-packages/django/db/backends/base/schema.py:250: in create_model
    to_column = field.remote_field.model._meta.get_field(field.remote_field.field_name).column

Actual error is in get_field() at

/tmp/deleteme/local/lib/python2.7/site-packages/django/db/models/options.py:582: FieldDoesNotExist

The migration that appears to be causing the problems (wish I could tell which step was failing):

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
        ('spud', '0003_auto_20141217_0908'),
    ]

    operations = [
        migrations.AlterModelOptions(
            name='photo',
            options={'ordering': ['datetime', 'id']},
        ),
        migrations.RenameField(
            model_name='album',
            old_name='album_id',
            new_name='id',
        ),
        migrations.RenameField(
            model_name='category',
            old_name='category_id',
            new_name='id',
        ),
        migrations.RenameField(
            model_name='person',
            old_name='person_id',
            new_name='id',
        ),
        migrations.RenameField(
            model_name='photo',
            old_name='photo_id',
            new_name='id',
        ),
        migrations.RenameField(
            model_name='place',
            old_name='place_id',
            new_name='id',
        ),
    ]

[Brian May]

Code that causes this failure on ​github.

[Brian May]

Looks like the above migration fails for sqlite (above error), fails for mysql (different error), but works for postgresql. So to try and summarize:

I said earlier this works for Python3.4, doesn't appear to be the case any more; it fails regardless of Python version. So maybe I missed up the earlier test.

So to get this fail:

• Use mysql (different error) OR
• Use sqlite and Django >= 1.9 (error quoted above)

Postgresql appears to work with any Django version. So maybe I somehow accidentally made the migration postgresql specific???

Test results available here: ​https://travis-ci.org/brianmay/spud/builds/117384657

[Brian May]

Suspect the failure on mysql is that mysql doesn't like having the primary key renamed.

The failure with sqlite looks like some sort of regression in Django 1.9. No idea if is it feasible to fix or not.

Apologies if any of this was documented somewhere and I missed it...

I plan to work around this for now by squashing the migrations and removing the rename.

[Tim Graham]

Bisected the behavior change to 720ff740e70e649a97fcf0232fec132569a52c7e. Haven't confirmed it's a bug in Django and not in the application but accepting for further investigation.

If you could create a more minimal set of steps to reproduce the issue, that would be helpful.

[Tim Graham]

I'm attaching a minimal project to reproduce. Add the rename app to a project and run python manage.py test rename and you should see FieldDoesNotExist: Foo has no field named u'album_id' (reproducible with SQLite only). If someone can propose a fix, I don't mind spending more time trying to write a regression test but as I'm not sure where the fix lies, I'm not sure at what level to write a test.

[Tim Graham]

Alex, could you investigate as you're the author on the patch where the behavior changed?

[Alex Hill]

Sure. I'm away for Easter weekend with just phone, I'll have a look next week.

I reckon it will have something to do with the issues described in ​https://github.com/django/django/pull/4122