AlterIndexTogether in historical migration drops overlapping indices

[Michael Herrmann]

1. Create a model with two index_together fields: ('a', 'b')
2. Run makemigrations.
3. Add field c and change index_together to (('a', 'b'), ('a', 'b', 'c')).
4. Run makemigrations.
5. Run migrate.

For at least SQLite, this silently drops the two-column index and only produces the three-column index.

I am attaching code that can be used to reproduce the issue. To run it, place all files in the same folder, cd into it and execute:

chmod +x steps.sh
./steps.sh

(This requires macOS or Linux. It should be easy to see how to execute the steps on Windows.)

Executing steps.sh with Django 5 produces error

ValueError: Found wrong number (0) of indexes for polls_mymodel(a, b).

On the other hand, executing the script with Django 4 succeeds without any errors.

(Please see the script for what I mean by "executing with Django 4/5".)

The script generates three migrations. And actually, the bug is already there after the second migration. But the third migration highlights the issue with an error message.

I am attaching script show_indexes.py that can be used to debug the issue. It shows the indexes in the SQLite database. When steps.sh is executed with Django 4 and without the third migration, then show_indexes.py displays two indexes - one for two and one for three columns. On the other hand, when steps.sh is executed with Django 5 (and without the third migration), then show_indexes.py only shows one index, for the three columns. show_indexes.py needs Django 4.

[Jacob Walls]

Thanks for the clear report. Bisected to 2abf417c815c20f41c0868d6f66520b32347106e. There was a follow-up to that removal with respect to fixing a crash in b44efdfe543c9b9f12690b59777e6b275cb08103, but it looks like we did not go as far as fixing what you report here: that the correct indexes are not recreated following a table remake.

It looks like Simon sketched out a possible implementation on the ​forum before a slimmer implementation was chosen to fix the TypeError:

We can definitely adapt AlterIndexTogether.state_forwards and .database_(forwards|backwards) to delegate to AddIndex and RemoveIndex by introspecting state and from_state though.

Whichever solution we decide to go forward with should serve as a validation for attempting the same deprecation path on unique_together -> constraints.

#31834 is almost identical for unique_together, but it's possible they could be (might have to be?) solved separately, since here we're talking about an operation that can only exist in historical migrations.

[nzioker]

I'll work on this fix. Based on Jacob's analysis, I'll implement the approach of adapting AlterIndexTogether to delegate to AddIndex and RemoveIndex operations.