Custom data migration for M2M with through_fields not working

[Brian Nettleton]

When writing a data migration for a model with a field which uses through fields the through_fields are not honored in the model retrieved from the migration's app registry.

Here is an example of such a data migration using the models from the Django documentation for through_fields. This data migration has an assert at the end of the forwards function which fails. Note that issuing the similar instructions in a Django shell works fine, the issue is specific to retrieving a model in a migration using apps.get_model. The models used and instructions for recreating the problem are also included below.

# Generated by Django 4.2.17 on 2025-01-02 19:35

from django.db import migrations


def forwards(apps, schema_editor):
    Person = apps.get_model('groups', 'Person')
    Group = apps.get_model('groups', 'Group')
    Group._meta.local_many_to_many[0].remote_field.through_field = ("group", "person")
    Membership = apps.get_model('groups', 'Membership')

    # Initialize some data in the database
    sally, _ = Person.objects.get_or_create(name="Sally Forth")
    steve, _ = Person.objects.get_or_create(name="Steve Smith")
    alice, _ = Person.objects.get_or_create(name="Alice Adams")
    grp1, _ = Group.objects.get_or_create(name="Group 1")
    grp2, _ = Group.objects.get_or_create(name="Group 2")
    admin, _ = Person.objects.get_or_create(name="Administrator")
    Membership.objects.get_or_create(
        group=grp1,
        person=sally,
        inviter=admin,
        invite_reason="Initial setup via migration"
    )
    Membership.objects.get_or_create(
        group=grp1,
        person=steve,
        inviter=admin,
        invite_reason="Initial setup via migration"
    )
    Membership.objects.get_or_create(
        group=grp1,
        person=alice,
        inviter=admin,
        invite_reason="Initial setup via migration"
    )

    # Okay, now also put everyone whose name starts with an "S" and is in Group 1 into Group 2
    for s_member in grp1.members.filter(name__startswith="S"):
        Membership.objects.get_or_create(
            group=grp2,
            person=s_member,
            inviter=admin,
            invite_reason="Initial setup 2: Put the initial 'S' people from Group 1 into Group 2"
        )

    print(f"\n{grp2.members.count()=}\n")
    assert grp2.members.count() == 2  # ===== FAILS =====


class Migration(migrations.Migration):

    dependencies = [
        ('groups', '0001_initial'),
    ]

    operations = [
        migrations.RunPython(forwards),
    ]

The models used for this migration are as follows.

from django.db import models


class Person(models.Model):
    name = models.CharField(max_length=50)


class Group(models.Model):
    name = models.CharField(max_length=128)
    members = models.ManyToManyField(
        Person,
        through="Membership",
        through_fields=("group", "person"),
    )


class Membership(models.Model):
    group = models.ForeignKey(Group, on_delete=models.CASCADE)
    person = models.ForeignKey(Person, on_delete=models.CASCADE)
    inviter = models.ForeignKey(
        Person,
        on_delete=models.CASCADE,
        related_name="membership_invites",
    )
    invite_reason = models.CharField(max_length=64)

Steps to reproduce:

1. Create a new empty project
2. Create a new empty app called groups and add to INSTALLED_APPS in project settings.py
3. Add models above to groups/models.py
4. Make initial migrations
5. Run initial migrations
6. Create empty migration for groups app
7. Put migration above into empty migration file
8. Run migrate to reproduce the problem

[Simon Charette]

I find it hard to believe but through_fields never actually worked in migrations since its introduction in c627da0ccc12861163f28177aa7538b420a9d310 (#14549) as ManyToManyField.deconstruct ​never special cased it. While the lack of support for through (which is by definition more common) was ​noticed during the introduction of migrations through_fields was likely missed as both features landed around the same time in the 1.7 release cycle.

Brian, given you've already spent time investigating where through_fields is stored (from your provided reproduction case) would you be interested in submitting a patch? Adjusting ManyToManyField.deconstruct to include through_fields if provided like so

django/db/models/fields/related.py

@@ -1772 +1772 @@
                 kwargs["through"] = self.remote_field.through
             elif not self.remote_field.through._meta.auto_created:
                 kwargs["through"] = self.remote_field.through._meta.label
+        if through_fields := getattr(self.remote_field, "through_fields", None):
+            kwargs["through_fields"] = through_fields
         # If swappable is True, then see if we're actually pointing to the target
         # of a swap.
         swappable_setting = self.swappable_setting

With extra assertions in ​the associated many-to-many field deconstruction test should do.

[Brian Nettleton]

Ok, my first Django pull request! ​https://github.com/django/django/pull/19006

Simon, thanks for the pointers on where to fix this.

[Sarah Boyce <42296566+sarahboyce@…>]

In b13b868:

Fixed #36061 -- Added migration support for ManyToManyField.through_fields.

Added through_fields support to ManyToManyField.deconstruct.
Thanks to Simon Charette for pointers and the review.

[Sarah Boyce <42296566+sarahboyce@…>]

In b96e4c04:

[5.2.x] Fixed #36061 -- Added migration support for ManyToManyField.through_fields.

Added through_fields support to ManyToManyField.deconstruct.
Thanks to Simon Charette for pointers and the review.

Backport of b13b8684a04d0bc1081104c5973c62c27dc673b0 from main.

[Tales Lelo da Aparecida]

Hi,

Could we have this mentioned in the release notes?

(I'm looking for the right place to post about this, but meanwhile here looked like a great start)

[Simon Charette]

Could we have this mentioned in the release notes?

No, Django doesn't document bug fixes so if you want to refer to the changes in a post you can either link to this ticket or the ​associated MR.