Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#14427 closed (fixed)

Add options to runtests.py to discover test pairings that fail

Reported by: russellm Owned by: nobody
Component: Testing framework Version: 1.2
Severity: Keywords:
Cc: Triage Stage: Design decision needed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Although unittests and doctests are supposed to be self contained, sometimes they have leaky side effects. For example, if a test case activates a language setting, but doesn't deactivate it, subsequent tests will be run in the wrong locale, which can cause tests to fail that pass when run in isolation.

Test failures like this are painful to try and track down, so ./runtests.py should have some tools to help narrow down the cause.

Attachments (1)

bisect_and_pair.diff (7.9 KB) - added by russellm 5 years ago.
Patch implementing bisection and pairing for runtests.py

Download all attachments as: .zip

Change History (4)

Changed 5 years ago by russellm

Patch implementing bisection and pairing for runtests.py

comment:1 Changed 5 years ago by russellm

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Design decision needed

Attached patch adds two flags to runtests.py:

  • --bisect=test_client
  • --pair=test_client

--bisect does a bisection of the test suite looking for a test that causes the test_client test to fail

--pair runs the test_client test with every other test until a test failure is discovered.

comment:2 Changed 5 years ago by russellm

  • Resolution set to fixed
  • Status changed from new to closed

(In [14079]) Fixed #14427 -- Added --bisect and --pair flags to runtests.py, making it easier to find pairs of tests that fail when run together.

comment:3 Changed 5 years ago by russellm

(In [14081]) [1.2.X] Fixed #14427 -- Added --bisect and --pair flags to runtests.py, making it easier to find pairs of tests that fail when run together.

Backport of r14079 from trunk.

Note: See TracTickets for help on using tickets.
Back to Top