Running tests with --keepdb messy to flush.

[Jarek Glowacki]

We run tests with --keepdb flag by default. If there are changes that require dropping the test database, we run the testsuite once without the keepdb flag.
However this not only drops the db before tests, but also drops it after tests too.
So next time we run the testsuite with the --keepdb flag, the database is recreated and migrations are run once more (unnecessarily).

It feels very clunky.

It all works as initially intended in https://code.djangoproject.com/ticket/20550:

I'd like to implement a similar feature in Django. In the light of my first experience, I suggest a simpler API. I would add a single flag --keepdb. When this flag is set:

• at the beginning of the tests, if the database already exists, Django doesn't ask to delete it; it goes ahead and reuses it;
• at the end of the tests, Django doesn't drop the database.

The original ticket author mentioned they'd implemented the --nocreatedb and --nodropdb options in their own project, prior to --keepdb existing, so they didn't have this issue.
I like the simplification that django offers and am not proposing to split keepdb into two separate flags, however offering a flag that enables either of the following would greatly unclunk --keepdbs use:

• "Don't drop db after running tests." (when --keepdb is off). Something like --preservedb, --nodropdb, etc.
• "Drop db prior to running tests." (when --keepdb is on). Something like --refreshdb, etc.

Yes, it's an extra flag, but it doesn't work the same way as --nocreatedb and --nodropdb, because for the majority of runs users just need to pass one flag (--keepdb), rather than multiple.

Another solution could be to offer a separate mcommand called droptestdb or something similar -- it's sole role would be to clean up anything that was left by the last --keepdb, so that the next one can run with a fresh db.

Finally, changing behaviour on --keepdb so that it somehow _knows_ when to prompt to start from scratch when it feels there is something not lining up in the migrations could be another way to go. Solution would be very magic, and maybe not great at guessing when to prompt, but would save the user from having to learn any of the other proposed options.

In short, --keepdb is flawed in how it requires TWO rounds of dropping/creating the db and running migrations (running without keepdb, then with keepdb again) to refresh a database.
I've put forward a few ideas on how to address this, but none particularly compelling.

Any thoughts?

[Mariusz Felisiak]

I don't agree that --keepdb is clunky it works as intended.

If there are changes that require dropping the test database.

I think that's something unusual. Everything will work without an extra roundtrip if you'll control these changes with migrations and without dropping the test database. So I would say that an issue is in your workflow rather than in Django itself. Running tests wasn't designed to drop a test database when something terrible happens. You use mix of --keepdb on/off calls to do this, but it's not a "supported" flow.

Finally, changing behaviour on --keepdb so that it somehow _knows_ when to prompt to start from scratch when it feels there is something not lining up in the migrations could be another way to go. Solution would be very magic, and maybe not great at guessing when to prompt, but would save the user from having to learn any of the other proposed options.

I think you've already answered that it's not feasible by using words like it somehow _knows_ and very magic.

[Jarek Glowacki]

Sorry, I should've provided some examples for when dropping the test db might be necessary:

• Doctoring an existing migration on a local branch. eg. I run tests on a local branch, applying a migration i've just added. I notice the migration is wrong and wish to change it before pushing it upstream. Running the tests again won't pick up my change. And to my knowledge I can't just unapply the migration like i can with the migrate mcommand.
• Rebasing migrations. eg. I've got a stale PR with a migration on it. I rebase my work and rename the migration and its dependencies to sit at the end of the list. Local test db still thinks I've already applied said changes and falls over trying to apply them a second time. (Yes I could just add a merge migration here, but it's unnecessary clutter if the migration exists only on my PR. And i don't think faking migrations is possible here, like it is with migrate.)

Are such workflows out of the ordinary?

Anyway, if this still isn't sufficiently convincing, won't argue further.
Will just add some custom code to do it in our own product I guess..

Thanks for the quick triage!