DiscoverRunner doesn't always group tests by class

[Chris Jerdonek]

The following is an issue that has been present in the test runner since before I started working on this area of the code this year (e.g. since this commit: ​https://github.com/django/django/tree/ec0ff406311de88f4e2a135d784363424fe602aa ), but I hadn't taken time to report it until now.

Namely, if tests are provided to the test runner on the command-line that aren't already grouped by test-case class, then DiscoverRunner won't group them by test-case class.

Here is an example (class A, class B, then class A again):

$ ./tests/runtests.py test_runner.tests.DependencyOrderingTests.test_simple_dependencies test_runner.tests.ManageCommandTests.test_custom_test_runner test_runner.tests.DependencyOrderingTests.test_chained_dependencies -v3 

Testing against Django installed in '/.../django/django'
Importing application test_runner
Found 3 test(s).
Skipping setup of unused database(s): default, other.
System check identified no issues (0 silenced).
test_simple_dependencies (test_runner.tests.DependencyOrderingTests) ... ok
test_custom_test_runner (test_runner.tests.ManageCommandTests) ... ok
test_chained_dependencies (test_runner.tests.DependencyOrderingTests) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK

This can interfere with things like parallel tests, which assume the tests are grouped by class, as well as the new shuffling code.

This can be fixed in ​reorder_tests(), which already iterates over all tests, as it has logic to group tests by test type and remove duplicates.

[Chris Jerdonek]

PR: ​https://github.com/django/django/pull/14621

[Chris Jerdonek]

With the patch:

$ ./tests/runtests.py test_runner.tests.DependencyOrderingTests.test_simple_dependencies test_runner.tests.ManageCommandTests.test_custom_test_runner test_runner.tests.DependencyOrderingTests.test_chained_dependencies -v3

Testing against Django installed in '/.../django/django'
Importing application test_runner
Found 3 test(s).
Skipping setup of unused database(s): default, other.
System check identified no issues (0 silenced).
test_simple_dependencies (test_runner.tests.DependencyOrderingTests) ... ok
test_chained_dependencies (test_runner.tests.DependencyOrderingTests) ... ok
test_custom_test_runner (test_runner.tests.ManageCommandTests) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK

[Mariusz Felisiak <felisiak.mariusz@…>]

In 5848b3a1:

Fixed #32914 -- Prevented test --shuffle from skipping test methods.

"test --shuffle" skipped test methods when test classes were mixed.
This changes runner.py's reorder_tests() to group by TestCase class.

Regression in 90ba716bf060ee7fef79dc230b0b20644839069f.