Opened 52 minutes ago
Last modified 51 minutes ago
#37510 new Bug
Parallel test runner should report failure for work interrupted on a dead worker — at Initial Version
| Reported by: | Jacob Walls | Owned by: | |
|---|---|---|---|
| Component: | Testing framework | Version: | dev |
| Severity: | Release blocker | Keywords: | |
| Cc: | Ahmed Mohamed, jbligh | Triage Stage: | Unreviewed |
| Has patch: | no | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
When manually testing the fix for #27734 (06463d3c11b3ce132075283d5f9368a269d66343), we engineered parallel test workers to die by creating them with maxtasksperchild=1. Now I realize that all we did was engineer a failure after the work has been completed and reported back to the test suite.
What we should have tested was a worker that dies *before* it reports success. Before #27734, we would have a loud failure. After #27734, we recover gracefully, but without reporting the results of the tests that were underway. This would lead to a misleading successful run, or more likely, a hang while waiting for the result.
I think we can keep the work form #27734 to recover gracefully and finish the rest of the work, but only as long as we actually report a failure for the interrupted work.
Bug in 06463d3c11b3ce132075283d5f9368a269d66343.
-
tests/test_runner/test_parallel.py
diff --git a/tests/test_runner/test_parallel.py b/tests/test_runner/test_parallel.py index df8532d4c6..0f29dca018 100644
a b class ParallelTestSuiteTest(SimpleTestCase): 376 376 pts.run(test_result) 377 377 378 378 self.assertIn("ValueError: woops", test_result.errors[0][1]) 379 380 def test_worker_exit_fails_run(self): 381 suite = ParallelTestSuite([TestSuite([TestCase()])], processes=1) 382 suite.serialized_aliases = set() 383 result = TestResult() 384 with mock.patch("multiprocessing.Pool") as mock_pool: 385 pool = mock_pool.return_value.__enter__.return_value 386 pool._pool = [mock.Mock(pid=101, exitcode=17)] 387 pool.imap_unordered.return_value.next.side_effect = [ 388 multiprocessing.TimeoutError, 389 AssertionError("The runner kept waiting after a worker died."), 390 ] 391 suite.run(result) 392 393 msg = "--parallel was unable to run all tests due to dead workers." 394 self.assertIn(msg, result.errors[0][1])
AssertionError: The runner kept waiting after a worker died.