RemoveField.state_forwards() crashes with AttributeError: 'NoneType' object has no attribute 'is_relation'

[Logan Gunthorpe]

Hi,

I'm trying to upgrade my project (originally created on 1.7) to the latest release but there's an issue with my existing migration files not working. The backtrace is below.

I'll attache my migration files but it seems to be related to the migration removing the 'id' field which was automatically created and then removed in a future migration. Commenting out the RemoveField in the second migration seems to fix the issue but I'm not sure if it's entirely correct to do so.

Thanks,

Logan

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 163, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/db/migrations/executor.py", line 81, in _create_project_state
    migration.mutate_state(state, preserve=False)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/db/migrations/migration.py", line 92, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "/home/logang/projects/stove/software/website_dev/env/lib/python3.4/site-packages/django/db/migrations/operations/fields.py", line 148, in state_forwards
    delay = not old_field.is_relation
AttributeError: 'NoneType' object has no attribute 'is_relation'

[Tim Graham]

Could you provide a more minimal (e.g. not involve other apps like timezone_field) and complete example (including models)?

Ideally, you could also ​bisect to find the commit where the behavior changed.

[Markus Holtermann]

I'm not entirely sure how you got to that line delay = not old_field.is_relation with old_field being None as this means there's no field with that name in state.models[app_label, self.model_name_lower].fields, but I think it's safe to go with delay = old_field is None or not old_field.is_relation.

[Markus Holtermann]

As Tim suggested, it would be helpful to have your full migration history from the time the field was first added, up to this migration that fails. Including field and model renames, etc. If you could then slim that down to a minimal example that would be rad.

[Logan Gunthorpe]

Sorry for the delay, the bug tracker didn't pick up my email address so it never sent me any emails.

I've bisected to find this is the first bad commit:

45ded053b1f4 Fixed #27666 -- Delayed rendering of recursivly related models in migration operations.

I'll attach a bisect log.

So far, I have not been able to produce a minimal example. I'm not sure why that is.

Thanks,

Logan

[Logan Gunthorpe]

I'm not exactly sure how I got the exact migrations I have. I'm pretty sure I created the model with the onetoonefield, then set it as primary_key in a second migration (thus removing the id field).

However, trying to reproduce this with a minimal setup always seems to work. I wonder if I did something really stupid and remove the id field from the initial migration. It was so long ago I can't really say for sure. Adding it back in seems to fix my problem.

If you want to assume that's what happened close this report I'm fine with that as I have a reasonable fix.

Thanks,

Logan

[Tim Graham]

Closing as "needsinfo" absent steps to reproduce.

[Matthew Dennis]

Ive managed to run into this as well on Django 1.11.

I had merged 2 very diverged branches(git) which had conflicting migrations that couldn't be merged. I deleted 2 leaf migrations that had been made on the branch and was just going to run make migrate to create only the fields that were different. I guess you can't do that. Anyway long story short I never committed the merge and reset my head but this error persists even after the fact. I'm assuming its something to do with it thinking I'm at a point that I'm not/doesn't exist.

I can't give you guys solid re-creation steps(and I don't know a lot about migrations) to get to this point so I'm not asking for a fix here.
But if you're setting a variable to None and then populating it conditionally in a loop it shouldn't be assumed it will have a value once the loop is complete. I'm suggesting an assert or some sort of exception handling for this. Perhaps with a message suggesting they report their steps to create this scenario on here.

Seeing as the comment above implies that old_field is None or ... would be a valid solution I don't see why you wouldn't just put that in. Adding that to my local file allows me to proceed but because I don't know whats causing this I can't merge this branch for fear of breaking it for others.

Thanks

[Tim Graham]

I closed #28489 as a duplicate but it's also missing steps to reproduce.

[Python Force]

I have the same issue like Logan.

Hello,

Was going from Django version 1.10.7 to 1.11.4 and my migrations were not working. All my migrations were migrated before the upgrade. I went back to 1.10.7 and it was working again. Going back to 1.11.4 same error.

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 163, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/[...]/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 81, in _create_project_state
    migration.mutate_state(state, preserve=False)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/migration.py", line 92, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 150, in state_forwards
    delay = not old_field.is_relation
AttributeError: 'NoneType' object has no attribute 'is_relation'

When I look in the fields.py file I can see on the line 139 a function:

def state_forwards(self, app_label, state):
    new_fields = []
    old_field = None
    for name, instance in state.models[app_label, self.model_name_lower].fields:
        if name != self.name:
            new_fields.append((name, instance))
        else:
            old_field = instance
    state.models[app_label, self.model_name_lower].fields = new_fields
    # Delay rendering of relationships if it's not a relational field
    delay = not old_field.is_relation
    state.reload_model(app_label, self.model_name_lower, delay=delay)

Found out that it was crashing on 1 of the model fields that are not existing anymore in the database. I added a line to print the name of the model and field to find out what is causing the crash.

If old_field = None then it is calling a method on empty variable and it throws and error.

Added this line
print app_label + " " + self.model_name_lower + " " + self.name

def state_forwards(self, app_label, state):
    new_fields = []
    old_field = None
    print app_label + " " + self.model_name_lower + " " + self.name
    for name, instance in state.models[app_label, self.model_name_lower].fields:
        if name != self.name:
            new_fields.append((name, instance))
        else:
            old_field = instance
    state.models[app_label, self.model_name_lower].fields = new_fields
    # Delay rendering of relationships if it's not a relational field
    delay = not old_field.is_relation
    state.reload_model(app_label, self.model_name_lower, delay=delay)

The Output was

cars video subtitle_url
cars photos gear
cars news tags
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "[...]/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 163, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/[...]/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 81, in _create_project_state
    migration.mutate_state(state, preserve=False)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/migration.py", line 92, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "[...]/venv/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 150, in state_forwards
    delay = not old_field.is_relation
AttributeError: 'NoneType' object has no attribute 'is_relation'

That means it crashed on App[Cars] Model[News] and Field[Tags] - tags were not in the DB anymore long time ago. I found this migration:

migrations.RemoveField(
    model_name='news',
    name='tags',
),

And deleted it.

After that the migration ran 100% and I am fully functional on 1.11.4.

For the record the function in 1.10.7 is - and that is working

def state_forwards(self, app_label, state):
     new_fields = []
     for name, instance in state.models[app_label, self.model_name_lower].fields:
         if name != self.name:
             new_fields.append((name, instance))
     state.models[app_label, self.model_name_lower].fields = new_fields
     state.reload_model(app_label, self.model_name_lower)

1. fields.py

​https://github.com/django/django/blob/master/django/db/migrations/operations/fields.py

2. Who was working on that one

​https://github.com/django/django/commit/45ded053b1f4320284aa5dac63052f6d1baefea9

[Tim Graham]

Can you please provide a sample project that reproduces the problem?

[Python Force]

I cannot because it took me 2 days to find out what it was and now it is fixed. Problem was that the migration with the missing field in the database based on that function was coming empty and you calling a method on that which is throwing an error.

As Logan said "Seeing as the comment above implies that old_field is None or ... would be a valid solution I don't see why you wouldn't just put that in."

I have explained everything above in details that is a simple project and it happened precisely just like that. I am not sure how better I can describe it. I am not sure why you have changed that migration function if this one is working well...in 1.10.7.

def state_forwards(self, app_label, state):
     new_fields = []
     for name, instance in state.models[app_label, self.model_name_lower].fields:
         if name != self.name:
             new_fields.append((name, instance))
     state.models[app_label, self.model_name_lower].fields = new_fields
     state.reload_model(app_label, self.model_name_lower)

I know 1.11 is a new version but I do not think it would crash the system anyway - This

old_field = None

is causing the error. And as a good practice and good software engineering if it is None why you did not even put there

try:
except

at least. Logan and I gave you a hint, and it has happened to 2 people already within 3 months, so something must be going on in there.

[testvinder]

I have the same problem. I have a migration that is trying to remove a field which has already been deleted from the database. I have no problems running migrations with 1.10.7.
I would be nice if 1.11.4 could at least let the user know which migration is causing the error.

[Python Force]

Testvinder add this line to the code and in the console you'll see where it breaks and that will give you idea where to look.

print app_label + " " + self.model_name_lower + " " + self.name

def state_forwards(self, app_label, state):
    new_fields = []
    old_field = None
    print app_label + " " + self.model_name_lower + " " + self.name
    for name, instance in state.models[app_label, self.model_name_lower].fields:
        if name != self.name:
            new_fields.append((name, instance))
        else:
            old_field = instance
    state.models[app_label, self.model_name_lower].fields = new_fields
    # Delay rendering of relationships if it's not a relational field
    delay = not old_field.is_relation
    state.reload_model(app_label, self.model_name_lower, delay=delay)

[johanovic]

I had the same issue. In my case it had to with the fact that I had copied the pycache of the migrations generated with a different setup to production. Can you clear the cache and see if that makes a difference?

[Carel]

I'm not sure if this is a bug for me too or simply my lack of understanding when it comes to rolling ones own migrations, but I'm triggering the same response. I'm converting a legacy SQLite database (Approx. 20-30 Tables still working through it) to Django. There is a models.IntField (Milliseconds since Unix/Linux Epoch) which I want to alter to a models.DateTimeField. Djangos' automigrate was complaining when I switched the table from unmanaged to managed so I'm manually performing the following actions within a custom migration :

1. Create a new models.DateTimeField called "Temp"
2. Using migrations.runPython Iterate overall the records and convert all the data in the original field, "Modified", to datetime.datetime and assign it to new field "Temp"
3. Remove the original field "Modified" using migrations.RemoveField
4. Rename the temporary field "temp" to "Modified" using migrations.RenameField

The process fails at Step 3 where the field being removed triggers the error, AttributeError: 'NoneType' object has no attribute 'is_relation'. Originally I was using Django 1.11.3/4 but have bumped to 1.11.7 and have the same problem.

Based on the code posted by PythonForce it would seem that the loop is not assigning an instance to old_field, so where it tries to assign a value for delay it is breaking as there is no attribute is_relation for None/NoneType. Now it seems delay should be a boolean value so I would like to suggest one replace the line delay = not old_field.is_relation with delay = old_field is not None and not old_field.is_relation then if old_field is None delay is False and the relation short circuits before testing the attribute. If old_field turns out to be an instance then delay depends upon old_field.is_related, as it presently does.

def state_forwards(self, app_label, state):
    new_fields = []
    old_field = None
    print app_label + " " + self.model_name_lower + " " + self.name
    for name, instance in state.models[app_label, self.model_name_lower].fields:
        if name != self.name:
            new_fields.append((name, instance))
        else:
            old_field = instance
    state.models[app_label, self.model_name_lower].fields = new_fields
    # Delay rendering of relationships if it's not a relational field
    delay = not old_field.is_relation
    state.reload_model(app_label, self.model_name_lower, delay=delay)

My suggestions is based upon only this snippet of code and I am not aware of any other context that may be relevant. I just tried this on the development branch and am now getting django.core.exceptions.FieldDoesNotExist: Translatorcache has no field named 'lastModifiedTime' which I can only assume is an improvement.

P.s. If the manner in which I have posted does not meet some guideline, let me know I'll try to neaten it up or whatever as necessary.

[yuanxu]

I had the same issue. Agree with testvinder. In my case, I've delete some fields in db, and have RemoveField in migrations. In 1.10.7, It works fun and crash in 2.0.7.

I just modify the fields.py in line 145 with:

 delay = old_field is None or  not old_field.is_relation

It works! but I don't know if is there are other issues.

[Shaun Stanworth]

I have seen and resolved this issue -- I had a migration containing a RenameField followed by a AlterField on the same field. By moving the RenameField to the previous migration, I was able to replay the migrations safely.

So I think the issue is either: the application of rename states (state_forwards) has an error, or there is an AlterField.reduce method that tries to squash renames into an alteration.

[Mariusz Felisiak]

​PR

[Mariusz Felisiak]

Django 2.2 and 3.0 doesn't receive bugfixes anymore. Django 3.1+ raises KeyError (e.g. KeyError: 'field_3', KeyError: ('test_one', 'mymodel2')) for nonexistent fields or models in manually edited migrations which is acceptable, IMO.