Opened 4 years ago

Last modified 3 months ago

#32114 closed Cleanup/optimization

Workaround for subtest issue with parallel test runner — at Version 1

Reported by: Jordan Ephron Owned by: nobody
Component: Testing framework Version: 3.1
Severity: Normal Keywords: Test subtest parallel
Cc: Adam Johnson, Sarah Boyce, Sage Abdullah, David Wobrock Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Jordan Ephron)

Related to https://code.djangoproject.com/ticket/26942

Django's ParallelTestSuite requires that all test cases be
pickleable. When a SubTest fails with an exception, it's
pickled and shipped back to the main process. Unfortunately,
Django's TestCases aren't always pickleable (for instance,
they may contain an instance of django.test.Client which can't
be cleanly pickled if an exception was raised by a view)

When the following test case is run in a parallel environment,
the test runner is unable to serialize the exception:

class SubtestBugTestCase(TestCase):
    def test_subtest_bug(self):
        with self.subTest("I am the subtest"):
            self.not_pickleable = lambda: 0
            self.assertTrue(False)

# AttributeError: Can't pickle local object 'SubtestBugTestCase.test_subtest_bug.<locals>.<lambda>'

Luckily, Django's DiscoverRunner only actually cares about a
small subset of the fields on TestCase, and those fields are
all of pickleable types. (This is also true of the PyCharm
test runner, which we use)

We work around the subtest issue by wrapping up the subtest
as follows:

class TestCaseDTO:
    def __init__(self, test):
        self._m_id = test.id()
        self._m_str = str(test)
        self._m_shortDescription = test.shortDescription()
        if hasattr(test, "test_case"):
            self.test_case = TestCaseDTO(test.test_case)
        if hasattr(test, "_subDescription"):
            self._m_subDescription = test._subDescription()

    def _subDescription(self):
        """conforming to _SubTest"""
        return self._m_subDescription

    def id(self):
        return self._m_id

    def shortDescription(self):
        return self._m_shortDescription

    def __str__(self):
        return self._m_str


class SubtestSerializingRemoteTestResult(RemoteTestResult):
    def addSubTest(self, test, subtest, err):
        subtest = TestCaseDTO(subtest)
        super().addSubTest(test, subtest, err)


class OurRemoteTestRunner(RemoteTestRunner):
    resultclass = SubtestSerializingRemoteTestResult


class OurTestSuite(ParallelTestSuite):
    runner_class = OurRemoteTestRunner


class OurDiscoverRunner(DiscoverRunner):
    parallel_test_suite = OurTestSuite

Now, to be fair, it's possible that some test runners do care
about fields that aren't captured by TestCaseDTO, so I'm
not sure whether this is truly a universal solution, but this
issue was a significant impediment to parallelizing our tests.

Would it be worthwhile to have this behavior in Django core?

Change History (1)

comment:1 by Jordan Ephron, 4 years ago

Description: modified (diff)
Note: See TracTickets for help on using tickets.
Back to Top