Add migrations support for adding ManyToMany "through" model

[lorinkoz@…]

Adding a "through" model on a ManyToMany field make "makemigrations" fail with an explicit error stating that it is not possible. I have worked it around by renaming the field, creating the migration (so as to make Django believe it's a new field) and then renaming the field back to the original name and creating another migration (which is correctly acknowledged by Django as a simple "renaming"). I understand the implications of adding a through model from the migrations perspective, but maybe a more friendly output could be achieved. Is it even possible to create some sort of interaction, like when you create a field with not null and no default value, so as to ask the user if they want to destroy/create a new field??

[Tim Graham]

As Andrew noted in #22476, "as the error message says, you cannot alter an M2M without a through model into one with a through model, at least not in the current system." I'm not sure how feasible it will be to add support for thits, but tentatively accepting.

[Andrew Godwin]

Triaging this to "someday/maybe". It would be nice, but it's a complex thing to add.

[Marten Kenbeek]

PR: ​https://github.com/django/django/pull/4312

[Tim Graham]

Markus reviewed the PR.

[Simon Charette]

FWIW a workaround for the current state of things, as long as through._meta.db_table points to the previously automatically generated db_table, is to wrap the generated operations in a SeparateDatabaseAndState.

[Simon Charette]

I'm not actively working on this ticket so I'll deassign myself but I made some progress by using a different approach.

My idea was to teach the auto-detector to detect ManyToMany(through) addition where through is a model compatible with the previously auto-created model. When it detected this case it would use the questioner to ask the user whether or not the previous table should be reused. On a "yes" a AddManyToManyThrough operation that has a noop database_forwards but an appropriate state_forwards to deal with the ModelState insertion would be added to the operations. If the new db_table wasn't matching or fields were added then the appropriate AlterModelTable and AddField operations would be added after the AddManyToManyThrough as well.

[CreateModel('Foo'), AddManyToManyThrough('Foo', 'bars', ManyToMany('Bar', through='FooBar')] could be optimized into
[CreateModel('Foo'), CreateModel('FooBar', fks), AddField('Foo', 'bars', ManyToMany('Bar', through='FooBar')] and the normal chain of optimizations could then take place.

The work so far is available on this ​branch

[Carsten Agger]

As a developer bitten by this I'd like to second this motion: A customer changed their data model mid-project, and now we need to add fields to a many-to-many relationship & I'm looking at a) deleting all migrations and reinitializing the DB or b) make one of the workarounds here play without being too much of a hack.

I'd be grateful to be able to do this automatically one day, I think the use case is common enough.

[Mariusz Felisiak]

As far as I'm aware, this transformation is too fragile to be automated (see ​https://youtu.be/4QF9UHEFPCY). It's also ​documented how to do this with SeparateDatabaseAndState().