Altering unique_together still sometimes missing deleted fields

[Julian Andrews]

This is likely a missed edge case from #23614.

Steps to reproduce:

1. Create an app spam
2. Create a model Spam

   class Spam(models.Model):
       a = models.CharField(max_length=255)
       b = models.CharField(max_length=255)
       c = models.CharField(max_length=255)
   
       class Meta:
           unique_together = (
               ('a', 'b'),
               ('b', 'c'),
           )
   

3. Make migrations
4. Delete field Spam.c
5. Delete the second unique_together constraint (('b', 'c'))
6. Make migrations
7. Run migrations

Output:

django.core.exceptions.FieldDoesNotExist: Spam has no field named u'c'

Note that the bug doesn't occur if no unique togther constraints remain.

Manually reversing the order of the operations in the migration (to remove the index first) produces a functional migration.

[Akshesh Doshi]

This is happening because the RemoveField operation is called before calling the AlterUniqueTogether operation, resulting in the dropping of the field before the index.

In the autodetector.py, although the RemoveField operations are created before the AlterUniqueTogether operations but the ​`_sort_migrations` method corrects the order of these operations to bring AlterUniqueTogether before RemoveField. But the ​`_optimize_migrations` method again reverses this order in self.migrations (which is finally what is returned by the _detect_changes method) with no optimization as such.

This change is somewhat related to ​https://github.com/django/django/commit/e470f311d654267ec86f9a6325ec500345b9dff2

[Akshesh Doshi]

The patch temporarily fixes the issue (attaching for anyone who gets stuck due to this bug) but some other edge cases would still get left. The actual fix would involve some changes in migrations.optimizer or the reduce methods of AlterUniqueTogether or RemoveField operations AFAICS.

[Akshesh Doshi]

PR: ​https://github.com/django/django/pull/6926

[Tim Graham]

Unchecking "Has patch" due to "not the right approach" comment on the PR.

[Akshesh Doshi]

An initial implementation of the suggested approach - ​https://github.com/django/django/pull/7038

[Tim Graham]

I closed #27933, #28084, and #28366 as duplicates.

[Cliff Dyer]

This issue just bit me on django 1.11.4, and required a couple hours of debugging to sort out.

[Simon Charette]

Pretty sure this is a duplicate of #28862 as well. I'll close this one as since the other one has more context.