Reduce the set of apps seen by individual tests
|Reported by:||Anssi Kääriäinen||Owned by:||nobody|
|Cc:||marc.tamlyn@…||Triage Stage:||Ready for checkin|
|Has patch:||yes||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
Currently Django's test suite contains close to 1000 models. For every transaction test case we need to flush all the tables, then install contenttypes and permissions for every model. This means around 4000 inserted rows per single test in TransactionTestCase.
To speed Django's test suite we could require that TransactionTestCases always contain information of what applications they depend on. By doing this we only need to flush a couple of models.
For example using PostgreSQL running transactions + transactions_regress takes 4.5 seconds when non-needed applications are masked, and 55 seconds without masking. MySQL and transactions + transactions_regress is 3.6s compared to 70s. The difference increases when more models are added to the test run. For example MySQL with transactions, transatcions_regress and queries tests takes 220s vs 9.1s. [The times do not include database creation which itself takes a lot of time and isn't changed by the patch].
Full suite on SQLite takes 560s on master, 213s patched. If normal TestCases were isolated, too, the test suite would be still faster (around 160s). The reason for this is fixture loading - for any fixture loaded we go through all apps, then check for all combinations of postfixes (.json, .xml, ...), and compression types (.tar.gz, .bzip, ...). Then check if file with any name of any combination exists. This is unsurprisingly very expensive to do.
A proof of concept patch is available from https://github.com/akaariai/django/compare/isolated_apps.
The idea of isolating test apps from each other doesn't work that well for user's tests. The problem is that usually applications contain a lot of dependencies (for example any time you do client.login() you are depending on django.contrib.auth and django.contrib.sessions). So, forcing this to on for user apps can't be done. Opt-in is of course possible, but I am not sure documenting this at all is necessary.
The patch needs some more work. The biggest issues are usage of settings._ALWAYS_INSTALLED_APPS for detecting if isolation should happen, and use of TRUNCATE ... CASCADE when using PostgreSQL (this could be dangerous, as flush could cascade to unmanaged tables).
Change History (42)
comment:40 Changed 3 years ago by
|Severity:||Normal → Release blocker|
|Status:||closed → new|
|Triage Stage:||Ready for checkin → Accepted|
comment:41 Changed 3 years ago by
|Severity:||Release blocker → Normal|
|Status:||new → closed|
|Triage Stage:||Accepted → Ready for checkin|