Migrations fail when when a model's parent class is changed to abstract -- Wrong SQL being generated

[Sawan Vithlani]

Consider the following class hierarchy:

class TestCase(models.Model):
    name = models.CharField(max_length=10)

class TerminalTest(TestCase):
    pass

class BrowserTest(TestCase):
    pass

$ ./manage.py makemigrations testbed
Migrations for 'testbed':
  testbed\migrations\0001_initial.py:
    - Create model TestCase
    - Create model BrowserTest
    - Create model TerminalTest

$ ./manage.py migrate testbed
Operations to perform:
  Apply all migrations: testbed
Running migrations:
Applying testbed.0001_initial... OK

Now we want to make the parent an abstract class:

class TestCase(models.Model):
    name = models.CharField(max_length=10)

    class Meta:
        abstract = True

class TerminalTest(TestCase):
    pass

class BrowserTest(TestCase):
    pass

./manage.py makemigrations testbed
You are trying to add a non-nullable field 'id' to browsertest without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> 1
You are trying to add a non-nullable field 'name' to browsertest without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> ' '
You are trying to add a non-nullable field 'id' to terminaltest without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> 1
You are trying to add a non-nullable field 'name' to terminaltest without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> ' '

Migrations for 'testbed':
  testbed\migrations\0002_auto_20170315_1711.py:
    - Remove field testcase_ptr from browsertest
    - Remove field testcase_ptr from terminaltest
    - Add field id to browsertest
    - Add field name to browsertest
    - Add field id to terminaltest
    - Add field name to terminaltest
    - Delete model TestCase

$ ./manage.py migrate testbed
Operations to perform:
  Apply all migrations: testbed
Running migrations:
  Applying testbed.0002_auto_20170315_1600...Traceback (most recent call last):
  File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 337, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: near ")": syntax error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "C:\my-venv\lib\site-packages\django\core\management\__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "C:\my-venv\lib\site-packages\django\core\management\__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\my-venv\lib\site-packages\django\core\management\base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\my-venv\lib\site-packages\django\core\management\base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "C:\my-venv\lib\site-packages\django\core\management\commands\migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "C:\my-venv\lib\site-packages\django\db\migrations\migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "C:\my-venv\lib\site-packages\django\db\migrations\operations\fields.py", line 147, in database_forwards
    schema_editor.remove_field(from_model, from_model._meta.get_field(self.name))
  File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\schema.py", line 249, in remove_field
    self._remake_table(model, delete_fields=[field])
  File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\schema.py", line 199, in _remake_table
    self.quote_name(model._meta.db_table),
  File "C:\my-venv\lib\site-packages\django\db\backends\base\schema.py", line 112, in execute
    cursor.execute(sql, params)
  File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "C:\my-venv\lib\site-packages\django\db\utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "C:\my-venv\lib\site-packages\django\utils\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 337, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: near ")": syntax error

The problem is that the SQL query generated by this (​code block) in django/db/backends/sqlite3/schema.py:

 # Copy data from the old table into the new table
            field_maps = list(mapping.items())
            self.execute("INSERT INTO %s (%s) SELECT %s FROM %s" % (
                self.quote_name(temp_model._meta.db_table),
                ', '.join(self.quote_name(x) for x, y in field_maps),
                ', '.join(y for x, y in field_maps),
                self.quote_name(model._meta.db_table),
            ))

reads:

'INSERT INTO "testbed_browsertest" () SELECT  FROM "testbed_browsertest__old" '

This is a wrong query in SQLite syntax. Correct query is:

'INSERT INTO "testbed_browsertest" SELECT  * FROM "testbed_browsertest__old" '

This has been verified by monkey-patching the query generated and having the migration move to the next stage -- where it fails for a completely different reason (nothing to do with SQL). See bug 27941

[Sawan Vithlani]

Please see bug 27941 for related bug.

[Tim Graham]

Duplicate of #25247 and/or #24424.