Add migrations support for adding ManyToMany "through" model

[lorinkoz@…]

Adding a "through" model on a ManyToMany field make "makemigrations" fail with an explicit error stating that it is not possible. I have worked it around by renaming the field, creating the migration (so as to make Django believe it's a new field) and then renaming the field back to the original name and creating another migration (which is correctly acknowledged by Django as a simple "renaming"). I understand the implications of adding a through model from the migrations perspective, but maybe a more friendly output could be achieved. Is it even possible to create some sort of interaction, like when you create a field with not null and no default value, so as to ask the user if they want to destroy/create a new field??

[Tim Graham]

As Andrew noted in #22476, "as the error message says, you cannot alter an M2M without a through model into one with a through model, at least not in the current system." I'm not sure how feasible it will be to add support for thits, but tentatively accepting.

[Andrew Godwin]

Triaging this to "someday/maybe". It would be nice, but it's a complex thing to add.

[Marten Kenbeek]

PR: ​https://github.com/django/django/pull/4312

[Tim Graham]

Markus reviewed the PR.

[Simon Charette]

FWIW a workaround for the current state of things, as long as through._meta.db_table points to the previously automatically generated db_table, is to wrap the generated operations in a SeparateDatabaseAndState.

[Simon Charette]

I'm not actively working on this ticket so I'll deassign myself but I made some progress by using a different approach.

My idea was to teach the auto-detector to detect ManyToMany(through) addition where through is a model compatible with the previously auto-created model. When it detected this case it would use the questioner to ask the user whether or not the previous table should be reused. On a "yes" a AddManyToManyThrough operation that has a noop database_forwards but an appropriate state_forwards to deal with the ModelState insertion would be added to the operations. If the new db_table wasn't matching or fields were added then the appropriate AlterModelTable and AddField operations would be added after the AddManyToManyThrough as well.

[CreateModel('Foo'), AddManyToManyThrough('Foo', 'bars', ManyToMany('Bar', through='FooBar')] could be optimized into
[CreateModel('Foo'), CreateModel('FooBar', fks), AddField('Foo', 'bars', ManyToMany('Bar', through='FooBar')] and the normal chain of optimizations could then take place.

The work so far is available on this ​branch

[Carsten Agger]

As a developer bitten by this I'd like to second this motion: A customer changed their data model mid-project, and now we need to add fields to a many-to-many relationship & I'm looking at a) deleting all migrations and reinitializing the DB or b) make one of the workarounds here play without being too much of a hack.

I'd be grateful to be able to do this automatically one day, I think the use case is common enough.

[Mariusz Felisiak]

As far as I'm aware, this transformation is too fragile to be automated (see ​https://youtu.be/4QF9UHEFPCY). It's also ​documented how to do this with SeparateDatabaseAndState().

[Shai Berger]

Replying to Mariusz Felisiak:

As far as I'm aware, this transformation is too fragile to be automated (see ​https://youtu.be/4QF9UHEFPCY). It's also ​documented how to do this with SeparateDatabaseAndState().

Nobody had anything to say about this for three years before that comment, and 18 months after... maybe it's time to wontfix this ticket?