support subtests when running tests in parallel

[Chris Jerdonek]

When I try running unit tests using the --parallel option ​described here, I get the following error:

NotImplementedError: subtests aren't supported at this time

​Here is where this message originates in the code. Was this planned for a future version? Thanks.

[Aymeric Augustin]

I didn't implement support for subtests when I wrote the parallel test runner because Django doesn't use them, the primary use case was Django's own test suite, and IIRC it wasn't entirely obvious how to deal with them. A patch implementing that method would certainly be appreciated.

[Chris Jerdonek]

I got a proof of concept for this working in a branch on GitHub ​here.

The code change looks like the following:

def addSubTest(self, test, subtest, err):
    # Following unittest.TestResult.addSubTest()'s implementation in
    # Python 3.5, we don't do anything with subtests on success.
    # See here for the Django ticket to support subtests with the
    # --parallel test option:
    # https://code.djangoproject.com/ticket/26942
    if err is not None:
        self.check_picklable(test, err)
        self.events.append(('addSubTest', self.test_index, subtest, err))
        self.stop_if_failfast()

I can work on a regression test, etc, over time.

[Chris Jerdonek]

Do you have any thoughts or ideas on a good testing approach for this?

It has a few tricky aspects.

At the least, it seems like the tests should test the code paths of (1) a successful and (2) a failing subtest in a somewhat end-to-end fashion. The success case can easily be met simply by including a test containing a subtest (which would suffice assuming the tests are already being run in parallel mode).

But for the failing case, we don't want to interfere with the test run itself. In my first attempt at programmatically instantiating and running a ParallelTestSuite within a test case, I got the following error (because it was running a parallel test within a parallel test):

  File "/.../lib/python3.5/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children

There is also the question of whether and which of these tests should run even if the tests aren't being run in parallel mode.

[Chris Jerdonek]

I posted a pull request for this ​here.

[Tim Graham]

Comments for improvement are on the PR.

[Chris Jerdonek]

​PR updated.

[Tim Graham <timograham@…>]

In 42dcceba:

Fixed #26942 -- Added support for subtests during parallel testing.