migrations.RemoveField causes OperationalError "no such column" upon migration when db_index=True

[cessor]

Description

I encountered the following error with django 4.1 in my Gitlab CI/CD Pipeline. When I bumped django versions from 4.0.7 to 4.1. my pipeline broke during the testing stage; specifically during db migrations. I have not changed any other source code.

Steps to reproduce

Minimal example attached. Run make green to see that it works with 4.0.7, run make red to see that it does not work with 4.1. It will build and exercise a docker container which installs all dependencies in isolation and sets up an example django app and run migrations.

Manual steps:

0. Install django 4.1
1. Create a new project
2. Create an app
3. Install app in project
4. Create a model
5. Add field on model, set db_index=True
6. Make migrations: $ python manage.py makemigrations
7. Remove field from model
8. Make migrations: $ python manage.py makemigrations
9. Apply migrations: $ python manage.py migrate

The migration fails with the following error (for an app called web, with a model called Entity with a field called attribute for example):

Running migrations:
Applying contenttypes.0001_initial... OK
...
Applying sessions.0001_initial... OK
Applying web.0001_initial... OK
Applying web.0002_remove_entity_attribute...Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
 return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 357, in execute
 return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: error in index web_entity_attribute_d22c3fcb after drop column: no such column: attribute

Details

The above steps create a set of migrations where at the end a RemoveField migration is produced. Applying this migration fails for fields which had db_index=True. The example I attached uses a SlugField where db_index defaults to True, setting this parameter to False will apply the migration without this error.

I reproduced the error with the following field types: TextField, IntegerField, SlugField, CharField, URLField

[cessor]

Minimal Example

[Mariusz Felisiak]

I'm really puzzled. I cannot reproduce this issue manually, but make red crashes.

[cessor]

Replying to Mariusz Felisiak:

I'm really puzzled. I cannot reproduce this issue manually, but make red crashes.

Thank you for giving this a try. I investigated this some further and realized that my manual description is not precise enough to reproduce the error. The example I attached uses a SlugField, but the error does not occur with other field types, such as CharFields. I noticed that SlugFields set db_index=True by default (See Link 1), and could reproduce the bug with other fields when setting db_index=True. I will change the bug description accordingly.

Links:

• 1: SlugField: ​https://github.com/django/django/blob/main/django/db/models/fields/__init__.py#L2301

[fizaashraf37]

Regression test of the bug reported

[fizaashraf37]

The issue is valid. I have written the regression test of the exact test case. And found the first bad commit as below:

3702819227fd0cdd9b581cd99e11d1561d51cbeb is the first bad commit
commit 3702819227fd0cdd9b581cd99e11d1561d51cbeb
Author: Mariusz Felisiak <felisiak.mariusz@gmail.com>
Date:   Fri Feb 11 22:21:58 2022 +0100

    Refs #32502 -- Avoided table rebuild when removing fields on SQLite 3.35.5+.

    ALTER TABLE ... DROP COLUMN was introduced in SQLite 3.35+ however
    a data corruption issue was fixed in SQLite 3.35.5.

 django/db/backends/sqlite3/features.py |  2 ++
 django/db/backends/sqlite3/schema.py   | 10 ++++++++++
 tests/schema/tests.py                  | 18 ++++++++++++++++++
 3 files changed, 30 insertions(+)
bisect found first bad commit