Opened 3 years ago

Last modified 11 months ago

#32827 assigned Cleanup/optimization

Squashing migrations isn't very effective in complicated projects; rewrite docs to provide a preferred manual trimming process

Reported by: Mike Lissner Owned by: David Sanders
Component: Documentation Version: dev
Severity: Normal Keywords: migrations, squashmigration, documentation
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

I'm creating this ticket as a tracker, having already discussed this briefly on the mailing list:

​https://groups.google.com/g/django-developers/c/xpeFRpMTBZw/m/lDq78EedAwAJ

and having already created a PR for this:

​https://github.com/django/django/pull/14408

My impression is that to get this back on the review queue I need this ticket.

Change History (8)

comment:1 by Carlton Gibson, 3 years ago

Triage Stage: Unreviewed β†’ Accepted

OK, happy to review a suggestion.Thanks.

comment:2 by Carlton Gibson, 3 years ago

Patch needs improvement: set

Not sure that squashmigrations is not the preferred option for most/many projects. It works perfectly well for a lot of cases. πŸ€”
Probably _Automatically squash_ vs _Manually squash_ rather than _trim_ vs _squash_ (for me)
Comments on ​PR

Last edited 3 years ago by Carlton Gibson (previous) (diff)

comment:3 by Jacob Walls, 3 years ago

Component: Database layer (models, ORM) β†’ Documentation
Owner: changed from nobody to Mike Lissner
Status: new β†’ assigned

comment:4 by David Sanders, 11 months ago

@Carlton & Nessita:

I agree there needs to be a bit in the docs for this; Mike has stepped down as he doesn't have any time but I have some time:

  • I agree that it should be "manually squashing" vs "automatic squashing"
  • I've only tried squashmigrations twice but both times failed and ended up costing me quite a few hours that my boss wasn't too happy about :D
  • The first time there was this circular dependency nightmare that I couldn't escape
  • The second I had to write some custom serialisation to deal with django-pgtrigger & in the end it didn't work out (I don't remember all the details as this was a few months ago)
  • I decided to finally manually squash them, but using the "replaces" feature that squashmigrations uses – and it worked a treat plus it didn't take very long!
  • The key thing to remember is that you *must* note down all the non-elidable operations.
  • These are the steps I used (they'll need to be tailored for general public):
    • I rm'd all my migrations
    • ran makemigrations (giving it a distinct new initial name)
    • used git to restore all the rm'd migrations
    • copy all the filenames into the replaces attribute
    • copied all my non-elidable operations *plus any dependencies* into the new initial migration
    • wait until all deployments have been migrated out of the "replaces zone" then follow the rest of the instructions on the docs

Sounded like Jessamyn had some issues with missing the custom user model - I didn't encounter this issue even though my user model was in the app I squashed πŸ€”

I'd like to forward a PR if that's ok.

in reply to:  4 comment:5 by Natalia Bidart, 11 months ago

Replying to David Sanders:

@Carlton & Nessita:

Hello! Thanks for including me.

I agree there needs to be a bit in the docs for this; Mike has stepped down as he doesn't have any time but I have some time:

  • I agree that it should be "manually squashing" vs "automatic squashing"

I don't object to this though I don't understand exactly which scenario would be "automatic squashing": even when running manage.py squashmigrations arg1 arg2, the user has to perform manual steps afterwards (unless I'm missing something, which can be the case!)

What I mean is that, in my experience, squashing migrations is always a manual operation (beyond running the management command, that is). There is more or less steps, and one may or may not use the management command. So if you ask me I would organize the docs roughly like this:

  1. Squashing migrations

You can use the provided management command ... <details here for this option>.

If the above does not work or does not suit your needs for whatever reason (<examples>), you could instead:

<details for option 2>
<details for option 3>

I guess my point is that at most I would use labels to distinguish the options like: using the mgmt command, resetting, replacing.

  • I've only tried squashmigrations twice but both times failed and ended up costing me quite a few hours that my boss wasn't too happy about :D

I've did it multiple times are previous work but it required a non trivial amount of coordination involving migration landing freezes, deployment, squashing, migration unfreeze. We only did this when migrate was too slow when running tests.

  • The first time there was this circular dependency nightmare that I couldn't escape

Luckily we did not have this! But we did have a test check to ensure apps would not create circular dependencies, so I'm sure that helped.

  • The second I had to write some custom serialisation to deal with django-pgtrigger & in the end it didn't work out (I don't remember all the details as this was a few months ago)
  • I decided to finally manually squash them, but using the "replaces" feature that squashmigrations uses – and it worked a treat plus it didn't take very long!
  • The key thing to remember is that you *must* note down all the non-elidable operations.

:see_no_evil:

  • These are the steps I used (they'll need to be tailored for general public):
    • I rm'd all my migrations
    • ran makemigrations (giving it a distinct new initial name)
    • used git to restore all the rm'd migrations
    • copy all the filenames into the replaces attribute
    • copied all my non-elidable operations *plus any dependencies* into the new initial migration
    • wait until all deployments have been migrated out of the "replaces zone" then follow the rest of the instructions on the docs

I never tried this procedure!

Sounded like Jessamyn had some issues with missing the custom user model - I didn't encounter this issue even though my user model was in the app I squashed πŸ€”

I'd like to forward a PR if that's ok.

Yes please!

comment:6 by David Sanders, 11 months ago

Thanks nessita :)

Also I'm guilty of posting here then cross-posting in the forum as well where Carlton had replied: ​https://forum.djangoproject.com/t/doc-update-describe-steps-for-manually-squashing-migrations/20624

comment:7 by David Sanders, 11 months ago

Owner: changed from Mike Lissner to David Sanders

comment:8 by David Sanders, 11 months ago

Draft PR up for feedback on progress: ​PR

Last edited 11 months ago by Natalia Bidart (previous) (diff)
Note: See TracTickets for help on using tickets.
Back to Top