Opened 12 hours ago
Last modified 2 hours ago
#36170 new Cleanup/optimization
Running no-op migrations (verbose_name, ...) slow on sqlite
Reported by: | Klaas van Schelven | Owned by: | |
---|---|---|---|
Component: | Migrations | Version: | 5.1 |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Accepted | |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Running no-op migrations (verbose_name, help_text, choices) triggers DB-wide check_constraints on sqlite
This is because this is done as part of the DatabaseSchemaEditor's exit which is unconditionally called as part of apply_migration.
Because no-op migrations cannot be ignored by design (also here) this means that changing your help-text will lead to an unnecessary check of all constraints in the database (if no workarounds are employed, e.g. squashing with something else).
If we accept the principle of "no ignoring of fields on-detect" as per the issues mentioned above, we might still try to detect the fact that for any given field and DB nothing happened in practice (and hence no check_constraints is needed)
In particular: the call to check_constraints
was introduced as part of this commit, in which it is directly tied to a 5-step process.
Without claiming to have fully thought through all consequences: can't we pull the check closer to those 5 steps, or make it conditional to _remake_table
having been actually called between enter & exit?
Refs / background:
I'm doing this in the context of Bugsink, a Self-hosted Error Tracker; SQLite is a first level citizen (for production) in that tool, which means large databases can be expected. On a not all too big ~100K-row database I timed the above unnecessary check to take ~20 seconds. Scaling that up by 1 or 2 orders of magnitude would be my goal. Charitably assuming linear growth in time-complexity, you can see how this can quickly become really annoying.
this SO post provides tricks for avoiding the creation of such migrations in the first place (may be relevant context as long as the above isn't fixed yet).
Change History (3)
comment:1 by , 12 hours ago
Component: | Uncategorized → Migrations |
---|---|
Type: | Uncategorized → Cleanup/optimization |
comment:2 by , 9 hours ago
Triage Stage: | Unreviewed → Accepted |
---|
comment:3 by , 2 hours ago
I'm not sure I'd call it a "compromise", or simply a "solution" :-)
A quick test on both a real migration and a non-DB-altering one shows that this diff has the desired effect.
Interesting use case. I think it's worth a shot but we should definitely error on the side of caution here as to avoid causing subtle integrity violation regressions. That is the
PRAGMA foreign_key_check
might be necessary for other operations that SQLite added support for over the years such asADD FIELD
assuming it's a foreign key that can be populated with invalid data.I believe the following should cover all no-op migrations while avoiding the daunting task of figuring out for which operations are safe to skip foreign key checks for
django/db/backends/sqlite3/schema.py
Does that sound like a reasonable compromise?