Opened 9 years ago

Closed 9 years ago

#23742 closed New feature (fixed)

Option to run Django tests in reverse

Reported by: Wojtek Ruszczewski Owned by: nobody
Component: Testing framework Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The idea is to add a --reverse option to be able to more thoroughly check for test side-effects.

Django test suite currently has some tests that have side-effects causing rare failures in tests usually run before them (or successes in tests usually run after them?). Debugging such issues is somewhat cumbersome as the exact test order depends on the environment.

One example is assertNumQueries dependence on cache priming (to be handled in another ticket) causing test_group_permission_performance to fail (for instance just when the full suite is run with a database on a ramdisk and memcached enabled). With the branch you may try:

./runtests.py admin_views.tests.GroupAdminTest --reverse

or

./runtests.py --pair admin_views.tests.GroupAdminTest.test_group_permission_performance --reverse

to see a failure when run after fixtures tests.

Attachments (2)

reverse-results-sqlite.txt (87.1 KB ) - added by Wojtek Ruszczewski 9 years ago.
reverse-results-sqlite-updated.txt (39.8 KB ) - added by Wojtek Ruszczewski 9 years ago.

Download all attachments as: .zip

Change History (11)

comment:1 by Wojtek Ruszczewski, 9 years ago

Needs documentation: set
Needs tests: set

Some tickets that are probably (not verified) experiencing the assertNumQueries cache dependence: #10899, #17377, #20432, #23641.

by Wojtek Ruszczewski, 9 years ago

Attachment: reverse-results-sqlite.txt added

comment:2 by Wojtek Ruszczewski, 9 years ago

Needs documentation: unset
Needs tests: unset

Added some suite ordering tests and short docs for --bisect, --pair and --reverse.

Attached the output of failing tests from running the whole suite in reverse with the default settings. 94 in total, some could be probably helped, but some may be hard to avoid. The other ticket with assertNumQueries fix mentioned in the description is #23746.

comment:3 by Tim Graham, 9 years ago

Patch needs improvement: set
Triage Stage: UnreviewedAccepted

This seams reasonable and helpful. It would be better if the documentation for the existing runtests.py options was submitted separately. There is also some missing documentation for the new argument to DiscoverTestRunner.

comment:4 by Wojtek Ruszczewski, 9 years ago

Patch needs improvement: unset

Rebased the branch on top of the merged existing options documentation (thanks for improving it so much) and added documentation for the new DiscoverTestRunner argument.

Alternatively, we could implement some shuffling, for example like the "more random order" -r flag of the Python's suite.

comment:5 by Wojtek Ruszczewski, 9 years ago

Update: rebased the pull, split some line breaking fixes into another commit, moved the new test to a better place, made the option available with manage.py test and made it reverse test cases within test classes (as defined in testing docs), but leave the classes in the working order.

As noticed by Preston, there is actually another level of reversing to consider. First, reversing labels should always work. Secondly, Django sorts tests into django.TestCase, django.SimpleTestCase and unittest.TestCase, these groups shouldn't be probably reversed as the order is guaranteed in documentation. Thirdly, cases within groups should be independent. And lastly, test methods would better be independent too (though there can be some reasonable exceptions).

Last edited 9 years ago by Wojtek Ruszczewski (previous) (diff)

by Wojtek Ruszczewski, 9 years ago

comment:6 by Wojtek Ruszczewski, 9 years ago

Making the option respect the classes order decreased the number of failures to 44 (when running the whole suite in reverse); attached an updated list with some guesses as to what is the issue in each case -- it seems all should be fairly easy to fix now.

comment:7 by Preston Timmons, 9 years ago

This patch looks good to me.

comment:8 by Preston Timmons, 9 years ago

Triage Stage: AcceptedReady for checkin

comment:9 by Tim Graham <timograham@…>, 9 years ago

Resolution: fixed
Status: newclosed

In e22c64dfc0230134c0e9d9ae1d9a5c1589fc8b6e:

Fixed #23742 -- Added an option to reverse tests order.

This is useful for debugging side effects affecting tests that
are usually executed before a given test. Full suite and pair
tests sort cases more or less deterministically, thus some test
cross-dependencies are easier to reveal by reversing the order.

Thanks Preston Timmons for the review.

Note: See TracTickets for help on using tickets.
Back to Top