Opened 10 years ago
Closed 10 years ago
#23742 closed New feature (fixed)
Option to run Django tests in reverse
Reported by: | Wojtek Ruszczewski | Owned by: | nobody |
---|---|---|---|
Component: | Testing framework | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Ready for checkin | |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
The idea is to add a --reverse
option to be able to more thoroughly check for test side-effects.
Django test suite currently has some tests that have side-effects causing rare failures in tests usually run before them (or successes in tests usually run after them?). Debugging such issues is somewhat cumbersome as the exact test order depends on the environment.
One example is assertNumQueries
dependence on cache priming (to be handled in another ticket) causing test_group_permission_performance
to fail (for instance just when the full suite is run with a database on a ramdisk and memcached enabled). With the branch you may try:
./runtests.py admin_views.tests.GroupAdminTest --reverse
or
./runtests.py --pair admin_views.tests.GroupAdminTest.test_group_permission_performance --reverse
to see a failure when run after fixtures
tests.
Attachments (2)
Change History (11)
comment:1 by , 10 years ago
Needs documentation: | set |
---|---|
Needs tests: | set |
by , 10 years ago
Attachment: | reverse-results-sqlite.txt added |
---|
comment:2 by , 10 years ago
Needs documentation: | unset |
---|---|
Needs tests: | unset |
Added some suite ordering tests and short docs for --bisect
, --pair
and --reverse
.
Attached the output of failing tests from running the whole suite in reverse with the default settings. 94 in total, some could be probably helped, but some may be hard to avoid. The other ticket with assertNumQueries
fix mentioned in the description is #23746.
comment:3 by , 10 years ago
Patch needs improvement: | set |
---|---|
Triage Stage: | Unreviewed → Accepted |
This seams reasonable and helpful. It would be better if the documentation for the existing runtests.py
options was submitted separately. There is also some missing documentation for the new argument to DiscoverTestRunner
.
comment:4 by , 10 years ago
Patch needs improvement: | unset |
---|
Rebased the branch on top of the merged existing options documentation (thanks for improving it so much) and added documentation for the new DiscoverTestRunner
argument.
Alternatively, we could implement some shuffling, for example like the "more random order" -r flag of the Python's suite.
comment:5 by , 10 years ago
Update: rebased the pull, split some line breaking fixes into another commit, moved the new test to a better place, made the option available with manage.py test
and made it reverse test cases within test classes (as defined in testing docs), but leave the classes in the working order.
As noticed by Preston, there is actually another level of reversing to consider. First, reversing labels should always work. Secondly, Django sorts tests into django.TestCase
, django.SimpleTestCase
and unittest.TestCase
, these groups shouldn't be probably reversed as the order is guaranteed in documentation. Thirdly, cases within groups should be independent. And lastly, test methods would better be independent too (though there can be some reasonable exceptions).
by , 10 years ago
Attachment: | reverse-results-sqlite-updated.txt added |
---|
comment:6 by , 10 years ago
Making the option respect the classes order decreased the number of failures to 44 (when running the whole suite in reverse); attached an updated list with some guesses as to what is the issue in each case -- it seems all should be fairly easy to fix now.
comment:8 by , 10 years ago
Triage Stage: | Accepted → Ready for checkin |
---|
comment:9 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
Some tickets that are probably (not verified) experiencing the
assertNumQueries
cache dependence: #10899, #17377, #20432, #23641.