Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#14427 closed (fixed)

Add options to runtests.py to discover test pairings that fail

Reported by: Russell Keith-Magee Owned by: nobody
Component: Testing framework Version: 1.2
Severity: Keywords:
Cc: Triage Stage: Design decision needed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Although unittests and doctests are supposed to be self contained, sometimes they have leaky side effects. For example, if a test case activates a language setting, but doesn't deactivate it, subsequent tests will be run in the wrong locale, which can cause tests to fail that pass when run in isolation.

Test failures like this are painful to try and track down, so ./runtests.py should have some tools to help narrow down the cause.

Attachments (1)

bisect_and_pair.diff (7.9 KB) - added by Russell Keith-Magee 6 years ago.
Patch implementing bisection and pairing for runtests.py

Download all attachments as: .zip

Change History (4)

Changed 6 years ago by Russell Keith-Magee

Attachment: bisect_and_pair.diff added

Patch implementing bisection and pairing for runtests.py

comment:1 Changed 6 years ago by Russell Keith-Magee

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Triage Stage: UnreviewedDesign decision needed

Attached patch adds two flags to runtests.py:

  • --bisect=test_client
  • --pair=test_client

--bisect does a bisection of the test suite looking for a test that causes the test_client test to fail

--pair runs the test_client test with every other test until a test failure is discovered.

comment:2 Changed 6 years ago by Russell Keith-Magee

Resolution: fixed
Status: newclosed

(In [14079]) Fixed #14427 -- Added --bisect and --pair flags to runtests.py, making it easier to find pairs of tests that fail when run together.

comment:3 Changed 6 years ago by Russell Keith-Magee

(In [14081]) [1.2.X] Fixed #14427 -- Added --bisect and --pair flags to runtests.py, making it easier to find pairs of tests that fail when run together.

Backport of r14079 from trunk.

Note: See TracTickets for help on using tickets.
Back to Top