Opened 4 months ago

Closed 3 months ago

Last modified 3 months ago

#33673 closed Bug (needsinfo)

PostgreSQL removes constraints when removing columns.

Reported by: George Owned by: nobody
Component: Migrations Version: 3.2
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by George)

I’ve encountered an odd bug(?) in Django in our production code. Dropping a column manually in migrations *without* removing its constraints leads to Django being unaware those constraints have been removed, and auto-generating incorrect migrations.

A short example:

Creating the migrations for this schema:

class M():
    a = models.IntegerField(default=1)
    b = models.IntegerField(default=2)
    class Meta:
        constraints = [
            UniqueConstraint(
                name="uniq_ab_1”, fields=[“a”, “b”]
            )
        ]

Create the constraint uniq_ab_1 is Postgres as expected (when using the \d+ command on the table to visualize)

However, this manual migration will remove the constraint, due to one of its member columns being deleted, this is just standard Postgres behavior:

        migrations.RemoveField(
            model_name=“m”,
            name=“a”,
        ),
        migrations.AddField(
            model_name=“m”,
            name=“a”,
            field=models.IntegerField(default=6),
        ),

This migration runs just fine. I can even modify the field of M again and run a further migration. However, using \d+ reveals that the uniq_ab_1 constraint` is gone from the database.

The only way I found out about this behavior was by renaming the constraint to uniq_ab_2, then auto-generating migrations and getting the error:
django.db.utils.ProgrammingError: constraint "uniq_ab_1" of relation … does not exist. In other words, on a rename, Django become aware a rename was happening and tried to remove the constraint from the database, in spite of itt being gone for a few migrations.

This behavior is rather unexpected. I’d assume Django would either:

  1. Notice the code model differs from the database schema in the original migration (when the constraint gets inadvertently removed) and fail
  2. Notice that the constraint is missing in the next migration (e.g. when adding an arbitrary field to M) and try to add it back again.

As it stands it seems like this ghost constraint is observed by some migration operations and not others.

Is this a known bug in Django?
Is there a way to guard against this behavior?
Is this perfectly normal and I am doing something wrong?

Change History (4)

comment:1 Changed 4 months ago by George

Description: modified (diff)

comment:2 Changed 3 months ago by Mariusz Felisiak

Resolution: needsinfo
Status: newclosed
Summary: Django silently removes constraints when removing columns, then arbitrarily choses to include them in migrationsPostgreSQL removes constraints when removing columns.

Thanks for the report, however PostgreSQL silently drops constraints, not Django itself. Unfortunately when you create migrations manually you must be aware of database-specific behavior and handle them on your own in most of cases. As far as I'm aware described use case (removing column to add it immediately in the same migration) is really unusual and not worth additional complexity and a performance regression for all users. Noticing that constraint is missing would require inspecting db constraints which is always quite expensive.

Is there any reason you cannot handle this automatically with AlterField()?

comment:3 in reply to:  2 ; Changed 3 months ago by George

Replying to Mariusz Felisiak:

Thanks for the report, however PostgreSQL silently drops constraints, not Django itself. Unfortunately when you create migrations manually you must be aware of database-specific behavior and handle them on your own in most of cases. As far as I'm aware described use case (removing column to add it immediately in the same migration) is really unusual and not worth additional complexity and a performance regression for all users. Noticing that constraint is missing would require inspecting db constraints which is always quite expensive.

Is there any reason you cannot handle this automatically with AlterField()?

No particular reason, no, and in this case I agree the issue is we had a poorly written manual migration, fault is on us. I just wanted to make sure this is indeed the behavior.

Out of curiosity, does django inspect the database at all for migrations or is it 100% looking at the mdoel's code and comparing it with the code from all previous migrations?

comment:4 in reply to:  3 Changed 3 months ago by Mariusz Felisiak

Replying to George:

Out of curiosity, does django inspect the database at all for migrations or is it 100% looking at the mdoel's code and comparing it with the code from all previous migrations?

Django migrations inspects database in many places, but all of them slow things down and this scenario is so specific that IMO it's not worth additional guard.

Note: See TracTickets for help on using tickets.
Back to Top