Opened 15 hours ago
Last modified 14 hours ago
#35849 new Bug
ParallelTestSuite reports error occurred in arbitrary test instead of setUpClass
Reported by: | David Winiecki | Owned by: | |
---|---|---|---|
Component: | Testing framework | Version: | 5.0 |
Severity: | Normal | Keywords: | |
Cc: | David Winiecki | Triage Stage: | Unreviewed |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
When running tests in parallel, if an error occurs in a test suite (TestCase
) before the first test method runs, for example in setUpClass
, then the shell output incorrectly states that the error occurred in an arbitrary test method, rather than, for example, setUpClass
.
This is a problem because a user reading the test output may not notice the real error location (for example, setUpClass
) in the traceback and may not run the other tests in the test suite.
This is especially problematic if automation parses the error location to automatically re-run tests (to address flaky, intermittently failing tests), because the other tests in the suite may never be run.
Repro:
- Create a Django project for testing:
Instructions to create the project yourself are below. Or you can use this existing project in the bugdemo/
directory at https://github.com/dcki/django/blob/demo_report_test_error_bug/bugdemo/README.md
To create the project yourself:
- Install Python.
- Create a virtualenv and activate it:
mkdir my_project && cd my_project && python3 -m venv .venv && . .venv/bin/activate
- Install Django and create a Django project:
pip install django && django-admin startproject mysite .
- Create a tests directory and an
__init__.py
file inside:mkdir mysite/tests && touch mysite/tests/__init__.py
- Install tblib:
pip install tblib
- Add these tests to the project:
mysite/tests/test_things.py
from django.test import SimpleTestCase class AlwaysFailTest(SimpleTestCase): @classmethod def setUpClass(cls) -> None: super().setUpClass() try: raise Exception('Intentional error') except: super().tearDownClass() raise def test_should_pass_a(self) -> None: pass def test_should_pass_b(self) -> None: pass # Exists so that, when an attempt is made to run tests in parallel, and at # least two TestCases are specified, then tests actually run in parallel. # (As of this writing, Django runs tests serially if there is only one test # suite to run, even if `--parallel=2` is specified.) class AlwaysPassTest(SimpleTestCase): def test_should_pass_a(self) -> None: pass def test_should_pass_b(self) -> None: pass
- Run the tests:
python3 manage.py test --parallel=2
Expected output:
Found 4 test(s). System check identified no issues (0 silenced). E.. ====================================================================== ERROR: setUpClass (mysite.tests.test_things.AlwaysFailTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dwiniecki/.pyenv/versions/3.11.9/lib/python3.11/unittest/suite.py", line 166, in _handleClassSetUp setUpClass() File "/Users/dwiniecki/src/other/dcki-django/bugdemo2/mysite/tests/test_things.py", line 9, in setUpClass raise Exception('Intentional error') ^^^^^^^^^^^^^^^^^ Exception: Intentional error ---------------------------------------------------------------------- Ran 2 tests in 0.214s FAILED (errors=1)
Actual output:
Found 4 test(s). System check identified no issues (0 silenced). E.. ====================================================================== ERROR: test_should_pass_b (mysite.tests.test_things.AlwaysFailTest.test_should_pass_b) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dwiniecki/.pyenv/versions/3.11.9/lib/python3.11/unittest/suite.py", line 166, in _handleClassSetUp setUpClass() File "/Users/dwiniecki/src/other/dcki-django/bugdemo/mysite/tests/test_things.py", line 9, in setUpClass raise Exception('Intentional error') ^^^^^^^^^^^^^^^^^ Exception: Intentional error ---------------------------------------------------------------------- Ran 2 tests in 0.221s FAILED (errors=1)
Change History (5)
comment:2 by , 15 hours ago
Some of this work was done personally and some of it was done for my employer https://www.techsmart.codes/
We don't believe we've submitted a "Corporate Contributor License Agreement" and I haven't submitted an "Individual Contributor License Agreement". Another team member at TechSmart intends to submit a CCLA in a few days or so.
comment:3 by , 15 hours ago
Cc: | added |
---|
comment:4 by , 14 hours ago
I think this bug has existed since ParallelTestSuite was first written. I've confirmed that it exists at least as far back as 4.2. (But personally I don't feel like it's important enough to backport to earlier versions of Django.)
As far as I've seen there is no related documentation to update.
comment:5 by , 14 hours ago
I'm tempted to check the "has patch" box but I'm not sure if it's too early to do that. :)
I wrote a proposed fix and regression test for this bug in a fork of the django GitHub project:
https://github.com/dcki/django/tree/ticket_35849
I want to help fix this but I'm not certain what the next step is. I read all the contributing documentation I could find.
tox
passes. (Excepttox -e docs
due to a problem withpyenchant
on my specific computer, butpip install pyenchant==3.3.0rc1 && cd docs && make spelling
passes.)