Option to run Django tests in reverse

[Wojtek Ruszczewski]

The idea is to add a --reverse option to be able to more thoroughly check for test side-effects.

Django test suite currently has some tests that have side-effects causing rare failures in tests usually run before them (or successes in tests usually run after them?). Debugging such issues is somewhat cumbersome as the exact test order depends on the environment.

One example is assertNumQueries dependence on cache priming (to be handled in another ticket) causing test_group_permission_performance to fail (for instance just when the full suite is run with a database on a ramdisk and memcached enabled). With the ​branch you may try:

./runtests.py admin_views.tests.GroupAdminTest --reverse

or

./runtests.py --pair admin_views.tests.GroupAdminTest.test_group_permission_performance --reverse

to see a failure when run after fixtures tests.

[Wojtek Ruszczewski]

Some tickets that are probably (not verified) experiencing the assertNumQueries cache dependence: #10899, #17377, #20432, #23641.

[Wojtek Ruszczewski]

Added some suite ordering tests and short docs for --bisect, --pair and --reverse.

Attached the output of failing tests from running the whole suite in reverse with the default settings. 94 in total, some could be probably helped, but some may be hard to avoid. The other ticket with assertNumQueries fix mentioned in the description is #23746.

[Tim Graham]

This seams reasonable and helpful. It would be better if the documentation for the existing runtests.py options was submitted separately. There is also some missing documentation for the new argument to DiscoverTestRunner.

[Wojtek Ruszczewski]

Rebased ​the branch on top of the merged existing options documentation (thanks for improving it so much) and added documentation for the new DiscoverTestRunner argument.

Alternatively, we could implement some shuffling, for example like the "more random order" ​-r flag of the Python's suite.

[Wojtek Ruszczewski]

Update: rebased ​the pull, split some line breaking fixes into another commit, moved the new test to a better place and made the option reverse test cases within test classes (as defined in ​testing docs), but leave the classes in the working order.

As ​noticed by Preston, there is actually another level of reversing to consider. First, reversing labels should always work. Secondly, Django sorts tests into django.TestCase, django.SimpleTestCase and unittest.TestCase, these groups shouldn't be probably reversed as the order is guaranteed in documentation. Thirdly, cases within groups should be independent. And lastly, test methods would better be independent too (though there can be some reasonable exceptions).

[Wojtek Ruszczewski]

Making the option respect the classes order decreased the number of failures to 44 (when running the whole suite in reverse); attached an updated list with some guesses as to what is the issue in each case -- it seems all should be fairly easy to fix now.

[Preston Timmons]

This patch looks good to me.

[Tim Graham <timograham@…>]

In e22c64dfc0230134c0e9d9ae1d9a5c1589fc8b6e:

Fixed #23742 -- Added an option to reverse tests order.

This is useful for debugging side effects affecting tests that
are usually executed before a given test. Full suite and pair
tests sort cases more or less deterministically, thus some test
cross-dependencies are easier to reveal by reversing the order.

Thanks Preston Timmons for the review.