Opened 5 years ago

Closed 3 years ago

#12408 closed Bug (fixed)

test runner needs to reorder tests more explicitly

Reported by: sebleier@… Owned by: nobody
Component: Testing framework Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: yes
Needs tests: yes Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Right now the test runner partitions the tests into two bins, either the TestCase bin or the catch all bin. The ordering of the catch all bin then depends on the order in which the tests are seen. The problem is that TransactionTestCase tests sometimes write to the test database without cleaning up after itself and that can sometimes affect a doctest that runs right after it since the doctest doesn't flush the database before it runs. Granted, doctests leave themselves vulnerable to each other since they can modify the database and don't flush before or after they complete. I think adding a little more protection by specifying that doctests should run before TransactionTestCase tests would help a little for the case when people write TransactionTestCase tests that modify data that later doctests are not expecting.

I suspect this won't be an issue once a resolution of #5624 is found.

Attachments (1)

test_ordering.diff (986 bytes) - added by sebleier@… 5 years ago.

Download all attachments as: .zip

Change History (7)

Changed 5 years ago by sebleier@…

comment:1 Changed 5 years ago by ericholscher

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 4 years ago by mattmcc

  • Severity set to Normal
  • Type set to Bug

comment:3 Changed 4 years ago by julien

  • Needs documentation set
  • Needs tests set

comment:4 Changed 3 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:5 Changed 3 years ago by aaugustin

  • Easy pickings unset

Change Easy pickings from NULL to False.

comment:6 Changed 3 years ago by Ramiro Morales <cramm0@…>

  • Resolution set to fixed
  • Status changed from new to closed

In [f758bdab5eec3e615598948dd5bcf9bb7b910c9d]:

Fixed #18271 -- Changed stage at which TransactionTestCase flushes DB tables.

Previously, the flush was done before the test case execution and now
it is performed after it.

Other changes to the testing infrastructure include:

  • TransactionTestCase now doesn't reset autoincrement sequences either (previous behavior can achieved by using reset_sequences.) With this, no implicit such reset is performed by any of the provided TestCase classes.
  • New ordering of test cases: All unittest tes cases are run first and doctests are run at the end.

THse changes could be backward-incompatible with test cases that relied
on some kind of state being preserved between tests. Please read the
relevant sections of the release notes and testing documentation for
further details.

Thanks Andreas Pelme for the initial patch. Karen Tracey and Anssi
Kääriäinen for the feedback and Anssi for reviewing.

This also fixes #12408.

Note: See TracTickets for help on using tickets.
Back to Top