Adding explicit primary key with different type doesn't update related ManyToManyFields.

[STANISLAV LEPEKHOV]

Hello!
When i changed this models:

class StoreChain(models.Model):
    places = models.ManyToManyField(Place, blank=True)

class Place(models.Model):
    pass

to this edition (set pk with uuid type):

class StoreChain(models.Model):
    uid = models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, unique=True)
    places = models.ManyToManyField(Place, blank=True)

class Place(models.Model):
    uid = models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, unique=True)

Django creates a migration file that affects only the model tables, while the relationship table remains unchanged, which will cause an error, because the data type of the _ID fields in it also needs to be changed:

[Mariusz Felisiak]

Thanks for the report. Please don't accept your own tickets.

[Bhuvnesh]

IMO solving ticket-29574 might also solve this issue.

[STANISLAV LEPEKHOV]

Replying to Mariusz Felisiak:
Thanks for the comments!
I wanted to note that the problem also occurs for the models.ForeignKey fields.
The migration succeeds (in the absence of links), and after adding the link, an exception is thrown.
I apologize in advance for the machine translation =)

Here is the code that will throw the exception:

class StoreChain(models.Model):
    places = models.ManyToManyField(Place, blank=True)

class Place(models.Model):
    pass

if change to this edition (set pk with uuid type) and add relation after:

class StoreChain(models.Model):
    uid = models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, unique=True)
    place = models.ForeignKey (Place, blank=True, null=True)

class Place(models.Model):
    uid = models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, unique=True)

[Aravind Swaminathan]

Does this problem occur only while using a uuid field or even with other type fields as primary key?

[STANISLAV LEPEKHOV]

Replying to Aravind Swaminathan:

Does this problem occur only while using a uuid field or even with other type fields as primary key?

I found an error when changing the type to UUID, I did not check with other types of fields.

[Bhuvnesh]

I tried to implement a solution in a ​draft PR . Any input is appreciable.

[Bhuvnesh]

STANISLAV, With a default uuid value you cannot insert previous values of id in the new table ? So you can only change pk to uuid field when the db is empty ? Correct me if i'm wrong.

[STANISLAV LEPEKHOV]

Replying to Bhuvnesh:

STANISLAV, With a default uuid value you cannot insert previous values of id in the new table ? So you can only change pk to uuid field when the db is empty ? Correct me if i'm wrong.

The essence of the problem is that I cannot correctly change the DATA TYPE for the pk field (including with an empty database). If I have an integer data type and there is a m2m field in another model containing pk of these objects, then when changing the data type to uuid, there will be no changes in the m2m table after migration (the type int, int will remain there)

[Bhuvnesh]

OK, so i did some more work on this and found that it is much more complicated than i thought it would be:

1. the naive solution could be to remake all the tables that are referencing to a model's pk but even that is not possible in MySql as we cannot drop a table if it is referenced by other table.(throws error in case when 2 models are related to each other)
2. the only solution i can think of (and tried) is deleting all related models and then recreating them with new uid field.
3. One more way to do this can be to turn off FOREIGN_KEYS_CHECKS and then remake table, but that can be risky and can lead to data inconsistencies.

So i'm deassigning myself in case someone has a better solution.

[Carol Naranjo]

STANISLAV I know this is old, but I could not replicate the issue by just altering the PK type. At least in my test with sqlite I see the many-to-many table updated as expected. Can you specify with which DB did you encountered the issue?

Here is what I tried.

1. Initial setup with initial models (tables already existing in DB but with empty rows):

class Place(models.Model):
    pass

class StoreChain(models.Model):
    pass
    places = models.ManyToManyField(Place, blank=True)

2. Update models

class Place(models.Model):
    id = models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True)

class StoreChain(models.Model):
    id = models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True)
    places = models.ManyToManyField(Place, blank=True)

3. makemigrations and migrate ==> I see all 3 tables updated with the migration file generated as follows:

import uuid
from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('store', '0001_initial'),
    ]

    operations = [
        migrations.AlterField(
            model_name='place',
            name='id',
            field=models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False),
        ),
        migrations.AlterField(
            model_name='storechain',
            name='id',
            field=models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False),
        ),
    ]

However, if I try renaming the column as well the migration file will be generated with RemoveField and AddField which at least with my setup using sqlite is causing another error when applying the migration, hoewever I would track that new finding as a separate issue in another ticket.

[AI да Стас!]

Hi! I used:

PostgreSQL + Django 4.0.3 + psycopg2-binary 2.9.1

[Carol Naranjo]

Thanks for the details. I was able to reproduce the exact issue in older versions of Django (including the reported 4.0.3), psycopg2-binary 2.9.11, and the latest PostgreSQL (18.3 as of today). Here are my findings:

Behavior with Django 4.0.3 to 5.2.12:
I confirmed that in these versions, the migration proceeds but results in an inconsistent database state: The relationship table is not updated to match the new primary key type and the foreign key is dropped in the many-to-many table.

Behavior with Django >= 6.0:
I tested the same scenario in the latest versions (6.0 and 6.1-dev), and the behavior has changed significantly. Instead of a silent failure when attempting to run this particular migration (which generates RemoveField and AddField operations for the primary key change), the database now explicitly blocks the destructive operation with an InternalError:

# (0.003) ALTER TABLE "issue34151_place" DROP COLUMN "id"; args=None; alias=default
# ...
# django.db.utils.InternalError: cannot drop column id of table issue34151_place 
# because other objects depend on it
# DETAIL:  constraint issue34151_storechain_place_id_97554049_fk_issue3415 on 
# table issue34151_storechain_places depends on column id of table issue34151_place
# HINT:  Use DROP ... CASCADE to drop the dependent objects too.

This is, in my opinion, the correct and expected behavior. By triggering a database-level error, Django prevents the creation of the 'broken' state described in the original report so I recommend closing this ticket as the behavior in the latest mainstream-supported versions is consistent with expected database integrity standards.

Additional Information / Hints for Users
In order to perform this type of migration (changing both the name and type of a primary key), here are a few considerations:

• Manual Migration Steps: An alternative way to apply this change would be writing a migration with RenameField and AlterType operations. However, be aware that for this specific example, PostgreSQL would still fail to cast a BigInt directly into a UUID. This is a database-level requirement for explicit casting, not a bug in Django. If for example the type was altered to a compatible format (such as a string/VarChar) instead of a UUID, the operation would execute correctly as expected.

• The Limits of Automated Migration Generation: Django migrations provide a 'best guess' of the operations required to keep tables and relationships consistent. However, according to the documentation “some of the more complex operations are not autodetectable and are only available via a hand-written migration” (See ​link ) . When these limitations are hit, manual migration adjustments are the standard procedure.

[Carol Naranjo]

Closing the ticket as it is not relevant anymore as discussed in the previous comment.

[Jacob Walls]

Hi Carol, thanks for the triage. Did you bisect the behavior change to a specific commit? I tried to check your triage result, but when running the changes described in comment:20, I get AlterField & AlterField as described there, not RemoveField & AlterField as described in comment:22. Also, I see no difference all the way back to 4.2. Let's keep this open until someone else can confirm your triage.

[Carol Naranjo]

Hi Jacob, let me clarify. The result in comment:22 is not by applying the changes in comment:20, but rather by applying the changes in the original report and in comment:6. (comment:20 was only my alternative attempt to reproduce the issue when I didn't know which database was used).

That is, the issue before django 6.0 was when changing both field type AND name in one single step, which generates a script to recreate the column, which, when applying, will result in a database in an inconsistent state regarding the foreign keys.

After django 6.0 the migration script is the same, but the execution will fail, which will at least leave the Postgres database in a consistent state. I didn't bisect the exact commit, but from the release notes I would assume it relates to: ​https://docs.djangoproject.com/en/6.0/releases/6.0/#database-backend-api --> "BaseDatabaseSchemaEditor and PostgreSQL backends no longer use CASCADE when dropping a column."

[Carol Naranjo]

Maybe a more accurate but verbose title for this issue would be "Inconsistent DB state after applying a migration that failed silently: Adding explicit primary key with different type doesn't update related ManyToManyFields."

---
Furthermore, and at this point this is not regarding this specific issue. While investigating the current one and testing with sqlite I faced the issue that I later described in the ticket #37006. For which I opened this PR ​https://github.com/django/django/pull/21073 and is waiting a second review round. In that PR, the autodetector will recognize the alter operations instead of the recreation, so if merged, that PR will also avoid running into this issue in the first place.