Code

Opened 5 years ago

Closed 5 years ago

Last modified 3 years ago

#11613 closed (fixed)

Add a --failfast argument to the testrunner

Reported by: Alex Owned by: kmtracey
Component: Testing framework Version: 1.0
Severity: Keywords:
Cc: kmike84@…, jukvalim, randy@…, rpbarlow Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

This would basically signal the testrunner to stop and report the failure as soon as the failure occured, rather than obligating the user to wait for the test suite to finis running to see the failure. This is useful while testing for regressions.

Attachments (2)

failfast_1.diff (7.3 KB) - added by jukvalim 5 years ago.
ticket_11613.patch (7.5 KB) - added by rpbarlow 5 years ago.
Alrighty, this patch will do the trick!

Download all attachments as: .zip

Change History (18)

comment:1 Changed 5 years ago by anonymous

  • Cc kmike84@… added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 5 years ago by Alex

  • Triage Stage changed from Unreviewed to Accepted

comment:3 follow-up: Changed 5 years ago by drozzy

I think simple argument of "--verbosity=2" will let you know which tests failed.
Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

comment:4 in reply to: ↑ 3 Changed 5 years ago by kmtracey

Replying to drozzy:

I think simple argument of "--verbosity=2" will let you know which tests failed.

Yes, that will show you what failed (if you happen to notice it before it scrolls off the screen), but you don't get the failure details until the end of the run. If you are running the whole suite that may take a while. If you Ctrl-C the run after you see the first failure you still don't get the failure information so you then have to run just that test again to see if it fails in isolation...plus with this approach you have to be watching the test run like a hawk to catch the first failure. It would be more convenient if there was an option for the run to stop and print the details of the failure once the first failure is encountered. Then you can start the run and periodically check on it to see how it is going -- you don't have to watch it in detail and so long as it keeps running you know you are in good shape.

Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

That's why this is proposed as an option. It does certainly make sense when manually running the suite to check for regressions due to a code change you just made.

Changed 5 years ago by jukvalim

comment:5 Changed 5 years ago by jukvalim

  • Cc jukvalim added
  • Has patch set

I've come across situations where something like this would have been useful. When there are 1000+ tests to run to check if anything is broken, getting some info about test failure as soon as it happens can save a lot of time.

comment:6 Changed 5 years ago by robmadole

Related post on django-developers group (there is a mention about the --failfast option).

comment:7 Changed 5 years ago by rpbarlow

  • Owner changed from nobody to rpbarlow
  • Status changed from new to assigned

w00t. This is very cool.

comment:8 Changed 5 years ago by rpbarlow

  • Cc randy@… added

Also created #12364, which is similar to this.

comment:9 Changed 5 years ago by rpbarlow

I checked out that patch. It looks good. The unit test didn't include an init.py or a models.py, so it wouldn't run. I created a new patch that includes these files. Thanks for the patch!

comment:10 Changed 5 years ago by rpbarlow

  • Cc rpbarlow added
  • Owner changed from rpbarlow to kmtracey
  • Status changed from assigned to new
  • Triage Stage changed from Accepted to Ready for checkin

Hey Karen, this one's ready for commit. Go failfast!

comment:11 Changed 5 years ago by kmtracey

  • Owner changed from kmtracey to rpbarlow
  • Triage Stage changed from Ready for checkin to Accepted

There's a problem in adding a new keyword argument to the test runner: existing alternate test runners won't be expecting the new arg. It would be good to figure out how to add this in a way that won't cause older runners to fail.

comment:12 Changed 5 years ago by rpbarlow

  • Patch needs improvement set
  • Status changed from new to assigned

if you have a function foo(arg1, arg2, ...), you can get it's args by foo.func_code.co_varnames. We can use this to find out if our arguments are accepted by the function we are about to pass them to to get around this problem.

Changed 5 years ago by rpbarlow

Alrighty, this patch will do the trick!

comment:13 Changed 5 years ago by rpbarlow

  • Owner changed from rpbarlow to kmtracey
  • Patch needs improvement unset
  • Status changed from assigned to new
  • Triage Stage changed from Accepted to Ready for checkin

Hey Karen, now the patch only will use the failfast flag if the test runner will take it. Thanks!

comment:14 Changed 5 years ago by kmtracey

  • Resolution set to fixed
  • Status changed from new to closed

(In [11843]) Fixed #11613: Added a failfast option for test running. Thanks jukvalim and Randy Barlow.

comment:15 Changed 5 years ago by kmtracey

(In [11846]) Added tet runner tests mistakenly left out of r11843. Refs #11613.

comment:16 Changed 3 years ago by jacob

  • milestone 1.2 deleted

Milestone 1.2 deleted

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.