Converting from concrete to abstract inheritance creates invalid migrations

[Stephen Finucane]

I currently have two models using concrete inheritance and would like to split these out into wholly separate tables, dropping the base table in the process. To do this, I have copied all columns from the base table into the "child" tables, and would now like to drop the remaining reference to the base table. To do this, I was hoping to (a) add a new auto-incrementing id column to the child tables that is not yet a primary key, (b) use a data migration to copy the data from the parent_ptr_id column to the id column, and (c) drop the parent_ptr_id column and switch id to be the primary key. However, I've noticed that regardless of what I do, I won't be able to run the migration or generate new migrations because of errors like the below.

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/core/management/commands/makemigrations.py", line 168, in handle
    migration_name=self.migration_name,
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/db/migrations/autodetector.py", line 43, in changes
    changes = self._detect_changes(convert_apps, graph)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/db/migrations/autodetector.py", line 128, in _detect_changes
    self.old_apps = self.from_state.concrete_apps
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/db/migrations/state.py", line 213, in concrete_apps
    self.apps = StateApps(self.real_apps, self.models, ignore_swappable=True)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/db/migrations/state.py", line 272, in __init__
    self.render_multiple([*models.values(), *self.real_models])
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/db/migrations/state.py", line 307, in render_multiple
    model.render(self)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/db/migrations/state.py", line 578, in render
    return type(self.name, bases, body)
  File "/tmp/django-error/.venv/lib/python3.7/site-packages/django/db/models/base.py", line 229, in __new__
    base.__name__,
django.core.exceptions.FieldError: Local field 'id' in class 'Child' clashes with field of the same name from base class 'Parent'.

I suspect this is because the state tracking is not accurately capturing the change in table bases, but I don't know how to override this without faking dropping the entire table and recreating it. I was hoping for an AlterModelBases API but there doesn't appear to be one and I'm not sure where to start adding one.

Note that I'm taking this approach because the actual model mapping to Child holds a lot of data, and I suspect creating a new Child2 model, copying everything across, and then dropping Parent and Child, would likely take many hours if not days to do. I would drop down to raw SQL for this but I've yet to figure out how to correctly update the foreign key constraints such that Django can keep track of them.

Minimal reproducer below.

Steps to reproduce

Create a new project and app. In the app, create the following models.py.

from django.db import models


class Parent(models.Model):
    pass


class Child(Parent):
    name = models.TextField(blank=True)

Create the initial migrations.

$ python manage.py makemigrations                                                                                                                                                       
Migrations for 'core':                                                                                                                                                                                             
  core/migrations/0001_initial.py                                                                                                                                                                                  
    - Create model Parent                                                                                                                                                                                          
    - Create model Child

Modify the Child model in models.py so that it no longer subclasses Parent.

from django.db import models


class Parent(models.Model):
    pass


class Child(models.Model):
    name = models.TextField(blank=True)

Create the new migrations. You'll be asked to provide a default value for the new id column. Provide something arbitrary since we'll be modifying this by hand later to add the data migration:

$ python manage.py makemigrations core                                                                                                                                                                             
You are trying to add a non-nullable field 'id' to child without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:                                
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py
Select an option: 1                                 
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> 0                                               
Migrations for 'core':                              
  core/migrations/0002_auto_20200302_1358.py
    - Remove field parent_ptr from child
    - Add field id to child

Attempt to check if there are any migrations necessary now. Things will blow up.

$ python manage.py makemigrations core
Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()                                          
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  ....

This is an issue on 1.11 and 3.0, and presumably everything inbetween.

[Mariusz Felisiak]

Marking as a duplicate of #23521.