Opened 4 years ago
Closed 4 years ago
#32351 closed Bug (invalid)
AlterField migration on ForeignKey column fails on fresh DB when referenced model has been subsequently modified
Reported by: | djangobugreport | Owned by: | nobody |
---|---|---|---|
Component: | Migrations | Version: | 3.1 |
Severity: | Normal | Keywords: | migration AlterField ForeignKey default on_delete |
Cc: | Simon Charette | Triage Stage: | Unreviewed |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
I have run into a specific sequence of migrations that work one by one but fail when applied in sequence. The sequence is as follows (generate and apply migrations after each below step to reproduce the bug):
- Create two models with one referencing the other like so:
class Foo(models.Model): hoge = models.IntegerField(unique = True) class Bar(models.Model): foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null = True)
- Change the ForeignKey field on the referencing model to pull a default instance from a python function:
class Foo(models.Model): hoge = models.IntegerField(unique = True) def get_default_foo(): foo, created = Foo.objects.get_or_create( hoge = 0 ); return foo.id class Bar(models.Model): #foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null = True) foo = models.ForeignKey(Foo, on_delete = models.SET_DEFAULT, default = get_default_foo)
- Add a new field to the referenced model:
class Foo(models.Model): hoge = models.IntegerField(unique = True) piyo = models.IntegerField(default = 0) def get_default_foo(): foo, created = Foo.objects.get_or_create( hoge = 0 ); return foo.id class Bar(models.Model): #foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null = True) foo = models.ForeignKey(Foo, on_delete = models.SET_DEFAULT, default = get_default_foo)
Now, delete the DB and try to recreate from scratch, and the migration created in the second step above will fail complaining that the column "piyo" does not exist. Evidently, the migration runtime is not correctly referencing the old version of Foo without the piyo field, but rather the current version in the source code. Here is my stack trace from Visual Studio:
Operations to perform: Apply all migrations: TheApp, admin, auth, contenttypes, sessions Running migrations: Applying TheApp.0001_initial... OK Applying TheApp.0002_auto_20210114_1330...Traceback (most recent call last): File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 413, in execute return Database.Cursor.execute(self, query, params) sqlite3.OperationalError: no such column: TheApp_foo.piyo The above exception was the direct cause of the following exception: Traceback (most recent call last): File "D:\sandbox\django\three\DjangoWebProject1\manage.py", line 17, in <module> execute_from_command_line(sys.argv) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line utility.execute() File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\__init__.py", line 395, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv self.execute(*args, **cmd_options) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\base.py", line 371, in execute output = self.handle(*args, **options) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\base.py", line 85, in wrapped res = handle_func(*args, **kwargs) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\commands\migrate.py", line 245, in handle fake_initial=fake_initial, File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\executor.py", line 117, in migrate state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\executor.py", line 147, in _migrate_all_forwards state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\executor.py", line 227, in apply_migration state = migration.apply(state, schema_editor) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\migration.py", line 124, in apply operation.database_forwards(self.app_label, schema_editor, old_state, project_state) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\operations\fields.py", line 236, in database_forwards schema_editor.alter_field(from_model, from_field, to_field) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 138, in alter_field super().alter_field(model, old_field, new_field, strict=strict) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\base\schema.py", line 572, in alter_field old_db_params, new_db_params, strict) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 360, in _alter_field self._remake_table(model, alter_field=(old_field, new_field)) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 200, in _remake_table 'default': self.quote_value(self.effective_default(new_field)) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\base\schema.py", line 303, in effective_default return field.get_db_prep_save(self._effective_default(field), self.connection) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\base\schema.py", line 282, in _effective_default default = field.get_default() File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\fields\related.py", line 960, in get_default field_default = super().get_default() File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\fields\__init__.py", line 831, in get_default return self._get_default() File ".\TheApp\models.py", line 11, in get_default_foo hoge = 0 File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 573, in get_or_create return self.get(**kwargs), False File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 425, in get num = len(clone) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 269, in __len__ self._fetch_all() File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 1308, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 53, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\sql\compiler.py", line 1156, in execute_sql cursor.execute(sql, params) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 98, in execute return super().execute(sql, params) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\utils.py", line 90, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 413, in execute return Database.Cursor.execute(self, query, params) django.db.utils.OperationalError: no such column: TheApp_foo.piyo
I cannot see any obvious way to work around this problem without diving into the internals of the migration engine. Similar migrations work fine, it's only this combination of changing "on_delete" and "default" in this way that I've seen cause the problem. Although this isn't personally blocking a release for me, I've marked this bug as "Release Blocker" as IMO it is fairly severe because it could easily block the update of a production database if the above migrations had to be applied in sequence. (It would be possible to rewind the commit history and apply the migrations one by one along with the accompanying source code, but this could potentially be a huge amount of work, especially if dealing with an automated deployment system, etc.).
I have attached a zip containing a test Visual Studio Django project with the code above. I simply created an empty project with the Visual Studio Django wizard, updated requirements.txt to the most recent version of Django (3.1.5), changed the parts of the default project that caused errors on Django 3, and then wrote the test case above. There's nothing Visual Studio centric about this problem so it should reproduce on any platform as far as I can tell.
If there is a workaround for this bug, please advise. Thank you.
Attachments (1)
Change History (3)
by , 4 years ago
Attachment: | migration_bug.zip added |
---|
comment:1 by , 4 years ago
Severity: | Release blocker → Normal |
---|
After some more tinkering I've found the following work around:
- Modify the "default" argument of the models.ForeignKey call in the second generated migration to be some invalid ID value, like -1:
field=models.ForeignKey(default=TheApp.models.get_default_foo, on_delete=django.db.models.deletion.SET_DEFAULT, to='TheApp.foo'), # change to field=models.ForeignKey(default=-1, on_delete=django.db.models.deletion.SET_DEFAULT, to='TheApp.foo'),
- Generate a new fourth migration, which will run after the "piyo" field has been added, and change the "default" setting back to the callback function.
This works for me although it will have to be done again every time the model Foo changes.
I have also changed the level of this ticket to Normal since I noticed your FAQ states that "Release blocker" is for things blocking a release of Django (I had assumed it was for things blocking the release of user projects).
comment:2 by , 4 years ago
Cc: | added |
---|---|
Resolution: | → invalid |
Status: | new → closed |
Thanks for this report, however IMO it's not a supported to use callables that create new objects in the Field.default. Django evaluates default
before applying a migration, in your case there is a mismatch between model and table states. To avoid this issue in the RunPython()
operation you can get the model from the versioned app registry, e.g. apps.get_model("theapp", "Foo")
. I'm not sure if it will work in the Field.default
.
minimal reproduction for ticket #32351 migration bug