ParallelTestSuite reports error occurred in arbitrary test instead of setUpClass

[David Winiecki]

When running tests in parallel, if an error occurs in a test suite (TestCase) before the first test method runs, for example in setUpClass, then the shell output incorrectly states that the error occurred in an arbitrary test method, rather than, for example, setUpClass.

This is a problem because a user reading the test output may not notice the real error location (for example, setUpClass) in the traceback and may not run the other tests in the test suite.

This is especially problematic if automation parses the error location to automatically re-run tests (to address flaky, intermittently failing tests), because the other tests in the suite may never be run.

Repro:

1. Create a Django project for testing:

Instructions to create the project yourself are below. Or you can use this existing project in the bugdemo/ directory at ​https://github.com/dcki/django/blob/demo_report_test_error_bug/bugdemo/README.md

To create the project yourself:

1. Install Python.
2. Create a virtualenv and activate it: mkdir my_project && cd my_project && python3 -m venv .venv && . .venv/bin/activate
3. Install Django and create a Django project: pip install django && django-admin startproject mysite .
4. Create a tests directory and an __init__.py file inside: mkdir mysite/tests && touch mysite/tests/__init__.py
5. Install tblib: pip install tblib
6. Add these tests to the project:

mysite/tests/test_things.py

from django.test import SimpleTestCase


class AlwaysFailTest(SimpleTestCase):
    @classmethod
    def setUpClass(cls) -> None:
        super().setUpClass()
        try:
            raise Exception('Intentional error')
        except:
            super().tearDownClass()
            raise

    def test_should_pass_a(self) -> None:
        pass

    def test_should_pass_b(self) -> None:
        pass


# Exists so that, when an attempt is made to run tests in parallel, and at
# least two TestCases are specified, then tests actually run in parallel.
# (As of this writing, Django runs tests serially if there is only one test
# suite to run, even if `--parallel=2` is specified.)
class AlwaysPassTest(SimpleTestCase):
    def test_should_pass_a(self) -> None:
        pass

    def test_should_pass_b(self) -> None:
        pass

2. Run the tests: python3 manage.py test --parallel=2

Expected output:

Found 4 test(s).
System check identified no issues (0 silenced).
E..
======================================================================
ERROR: setUpClass (mysite.tests.test_things.AlwaysFailTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dwiniecki/.pyenv/versions/3.11.9/lib/python3.11/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
  File "/Users/dwiniecki/src/other/dcki-django/bugdemo2/mysite/tests/test_things.py", line 9, in setUpClass
    raise Exception('Intentional error')
    ^^^^^^^^^^^^^^^^^
Exception: Intentional error

----------------------------------------------------------------------
Ran 2 tests in 0.214s

FAILED (errors=1)

Actual output:

Found 4 test(s).
System check identified no issues (0 silenced).
E..
======================================================================
ERROR: test_should_pass_b (mysite.tests.test_things.AlwaysFailTest.test_should_pass_b)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dwiniecki/.pyenv/versions/3.11.9/lib/python3.11/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
  File "/Users/dwiniecki/src/other/dcki-django/bugdemo/mysite/tests/test_things.py", line 9, in setUpClass
    raise Exception('Intentional error')
    ^^^^^^^^^^^^^^^^^
Exception: Intentional error

----------------------------------------------------------------------
Ran 2 tests in 0.221s

FAILED (errors=1)

[David Winiecki]

I wrote a proposed fix and regression test for this bug in a fork of the django GitHub project:

​https://github.com/dcki/django/tree/fix_report_test_error

I want to help fix this but I'm not certain what the next step is. I read all the contributing documentation I could find.

tox passes. (Except tox -e docs due to a problem with pyenchant on my specific computer, but pip install pyenchant==3.3.0rc1 && cd docs && make spelling passes.)

[David Winiecki]

Some of this work was done personally and some of it was done for my employer ​https://www.techsmart.codes/

We don't believe we've submitted a "Corporate Contributor License Agreement" and I haven't submitted an "Individual Contributor License Agreement". Another team member at TechSmart intends to submit a CCLA in a few days or so.

[David Winiecki]

I think this bug has existed since ParallelTestSuite was first written. I've confirmed that it exists at least as far back as 4.2. (But personally I don't feel like it's important enough to backport to earlier versions of Django.)

As far as I've seen there is no related documentation to update.

[David Winiecki]

I'm tempted to check the "has patch" box but I'm not sure if it's too early to do that. :)