Opened 52 minutes ago

Last modified 51 minutes ago

#37510 new Bug

Parallel test runner should report failure for work interrupted on a dead worker — at Initial Version

Reported by: Jacob Walls Owned by:
Component: Testing framework Version: dev
Severity: Release blocker Keywords:
Cc: Ahmed Mohamed, jbligh Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When manually testing the fix for #27734 (06463d3c11b3ce132075283d5f9368a269d66343), we engineered parallel test workers to die by creating them with maxtasksperchild=1. Now I realize that all we did was engineer a failure after the work has been completed and reported back to the test suite.

What we should have tested was a worker that dies *before* it reports success. Before #27734, we would have a loud failure. After #27734, we recover gracefully, but without reporting the results of the tests that were underway. This would lead to a misleading successful run, or more likely, a hang while waiting for the result.

I think we can keep the work form #27734 to recover gracefully and finish the rest of the work, but only as long as we actually report a failure for the interrupted work.

Bug in 06463d3c11b3ce132075283d5f9368a269d66343.

  • tests/test_runner/test_parallel.py

    diff --git a/tests/test_runner/test_parallel.py b/tests/test_runner/test_parallel.py
    index df8532d4c6..0f29dca018 100644
    a b class ParallelTestSuiteTest(SimpleTestCase):  
    376376            pts.run(test_result)
    377377
    378378        self.assertIn("ValueError: woops", test_result.errors[0][1])
     379
     380    def test_worker_exit_fails_run(self):
     381        suite = ParallelTestSuite([TestSuite([TestCase()])], processes=1)
     382        suite.serialized_aliases = set()
     383        result = TestResult()
     384        with mock.patch("multiprocessing.Pool") as mock_pool:
     385            pool = mock_pool.return_value.__enter__.return_value
     386            pool._pool = [mock.Mock(pid=101, exitcode=17)]
     387            pool.imap_unordered.return_value.next.side_effect = [
     388                multiprocessing.TimeoutError,
     389                AssertionError("The runner kept waiting after a worker died."),
     390            ]
     391            suite.run(result)
     392
     393        msg = "--parallel was unable to run all tests due to dead workers."
     394        self.assertIn(msg, result.errors[0][1])
AssertionError: The runner kept waiting after a worker died.

Change History (0)

Note: See TracTickets for help on using tickets.
Back to Top