RemoveField.state_forwards() crashes with AttributeError: 'NoneType' object has no attribute 'is_relation'

[Logan Gunthorpe]

Hi,

I'm trying to upgrade my project (originally created on 1.7) to the latest release but there's an issue with my existing migration files not working. The backtrace is below.

I'll attache my migration files but it seems to be related to the migration removing the 'id' field which was automatically created and then removed in a future migration. Commenting out the RemoveField in the second migration seems to fix the issue but I'm not sure if it's entirely correct to do so.

Thanks,

Logan

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 163, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/db/migrations/executor.py", line 81, in _create_project_state
    migration.mutate_state(state, preserve=False)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/db/migrations/migration.py", line 92, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/db/migrations/operations/fields.py", line 148, in state_forwards
    delay = not old_field.is_relation
AttributeError: 'NoneType' object has no attribute 'is_relation'

[Tim Graham]

Could you provide a more minimal (e.g. not involve other apps like timezone_field) and complete example (including models)?

Ideally, you could also ​bisect to find the commit where the behavior changed.

[Markus Holtermann]

I'm not entirely sure how you got to that line delay = not old_field.is_relation with old_field being None as this means there's no field with that name in state.models[app_label, self.model_name_lower].fields, but I think it's safe to go with delay = old_field is None or not old_field.is_relation.

[Markus Holtermann]

As Tim suggested, it would be helpful to have your full migration history from the time the field was first added, up to this migration that fails. Including field and model renames, etc. If you could then slim that down to a minimal example that would be rad.

[Logan Gunthorpe]

Sorry for the delay, the bug tracker didn't pick up my email address so it never sent me any emails.

I've bisected to find this is the first bad commit:

45ded053b1f4 Fixed #27666 -- Delayed rendering of recursivly related models in migration operations.

I'll attach a bisect log.

So far, I have not been able to produce a minimal example. I'm not sure why that is.

Thanks,

Logan

[Logan Gunthorpe]

I'm not exactly sure how I got the exact migrations I have. I'm pretty sure I created the model with the onetoonefield, then set it as primary_key in a second migration (thus removing the id field).

However, trying to reproduce this with a minimal setup always seems to work. I wonder if I did something really stupid and remove the id field from the initial migration. It was so long ago I can't really say for sure. Adding it back in seems to fix my problem.

If you want to assume that's what happened close this report I'm fine with that as I have a reasonable fix.

Thanks,

Logan

[Tim Graham]

Closing as "needsinfo" absent steps to reproduce.

[Matthew Dennis]

Ive managed to run into this as well on Django 1.11.

I had merged 2 very diverged branches(git) which had conflicting migrations that couldn't be merged. I deleted 2 leaf migrations that had been made on the branch and was just going to run make migrate to create only the fields that were different. I guess you can't do that. Anyway long story short I never committed the merge and reset my head but this error persists even after the fact. I'm assuming its something to do with it thinking I'm at a point that I'm not/doesn't exist.

I can't give you guys solid re-creation steps(and I don't know a lot about migrations) to get to this point so I'm not asking for a fix here.
But if you're setting a variable to None and then populating it conditionally in a loop it shouldn't be assumed it will have a value once the loop is complete. I'm suggesting an assert or some sort of exception handling for this. Perhaps with a message suggesting they report their steps to create this scenario on here.

Seeing as the comment above implies that old_field is None or ... would be a valid solution I don't see why you wouldn't just put that in. Adding that to my local file allows me to proceed but because I don't know whats causing this I can't merge this branch for fear of breaking it for others.

Thanks

[Tim Graham]

I closed #28489 as a duplicate but it's also missing steps to reproduce.

[Python Force]

I have the same issue like Logan.

Hello,

Was going from Django version 1.10.7 to 1.11.4 and my migrations were not working. All my migrations were migrated before the upgrade. I went back to 1.10.7 and it was working again. Going back to 1.11.4 same error.

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 163, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/[...]/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 81, in _create_project_state
    migration.mutate_state(state, preserve=False)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/migration.py", line 92, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 150, in state_forwards
    delay = not old_field.is_relation
AttributeError: 'NoneType' object has no attribute 'is_relation'

When I look in the fields.py file I can see on the line 139 a function:

def state_forwards(self, app_label, state):
    new_fields = []
    old_field = None
    for name, instance in state.models[app_label, self.model_name_lower].fields:
        if name != self.name:
            new_fields.append((name, instance))
        else:
            old_field = instance
    state.models[app_label, self.model_name_lower].fields = new_fields
    # Delay rendering of relationships if it's not a relational field
    delay = not old_field.is_relation
    state.reload_model(app_label, self.model_name_lower, delay=delay)

Found out that it was crashing on 1 of the model fields that are not existing anymore in the database. I added a line to print the name of the model and field to find out what is causing the crash.

If old_field = None then it is calling a method on empty variable and it throws and error.

Added this line
print app_label + " " + self.model_name_lower + " " + self.name

def state_forwards(self, app_label, state):
    new_fields = []
    old_field = None
    print app_label + " " + self.model_name_lower + " " + self.name
    for name, instance in state.models[app_label, self.model_name_lower].fields:
        if name != self.name:
            new_fields.append((name, instance))
        else:
            old_field = instance
    state.models[app_label, self.model_name_lower].fields = new_fields
    # Delay rendering of relationships if it's not a relational field
    delay = not old_field.is_relation
    state.reload_model(app_label, self.model_name_lower, delay=delay)

The Output was

cars video subtitle_url
cars photos gear
cars news tags
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 163, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/[...]/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 81, in _create_project_state
    migration.mutate_state(state, preserve=False)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/migration.py", line 92, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 150, in state_forwards
    delay = not old_field.is_relation
AttributeError: 'NoneType' object has no attribute 'is_relation'

That means it crashed on App[Cars] Model[News] and Field[Tags] - tags were not in the DB anymore long time ago. I found this migration:

migrations.RemoveField(
    model_name='news',
    name='tags',
),

And deleted it.

After that the migration ran 100% and I am fully functional on 1.11.4.

For the record the function in 1.10.7 is - and that is working

def state_forwards(self, app_label, state):
     new_fields = []
     for name, instance in state.models[app_label, self.model_name_lower].fields:
         if name != self.name:
             new_fields.append((name, instance))
     state.models[app_label, self.model_name_lower].fields = new_fields
     state.reload_model(app_label, self.model_name_lower)

1. fields.py

​https://github.com/django/django/blob/master/django/db/migrations/operations/fields.py

2. Who was working on that one

​https://github.com/django/django/commit/45ded053b1f4320284aa5dac63052f6d1baefea9