Opened 7 years ago

Closed 7 years ago

Last modified 5 years ago

#11613 closed (fixed)

Add a --failfast argument to the testrunner

Reported by: Alex Gaynor Owned by: Karen Tracey
Component: Testing framework Version: 1.0
Severity: Keywords:
Cc: kmike84@…, Jukka Välimaa, randy@…, Randy Barlow Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

This would basically signal the testrunner to stop and report the failure as soon as the failure occured, rather than obligating the user to wait for the test suite to finis running to see the failure. This is useful while testing for regressions.

Attachments (2)

failfast_1.diff (7.3 KB) - added by Jukka Välimaa 7 years ago.
ticket_11613.patch (7.5 KB) - added by Randy Barlow 7 years ago.
Alrighty, this patch will do the trick!

Download all attachments as: .zip

Change History (18)

comment:1 Changed 7 years ago by anonymous

Cc: kmike84@… added

comment:2 Changed 7 years ago by Alex Gaynor

Triage Stage: UnreviewedAccepted

comment:3 Changed 7 years ago by Andriy Drozdyuk

I think simple argument of "--verbosity=2" will let you know which tests failed.
Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

comment:4 in reply to:  3 Changed 7 years ago by Karen Tracey

Replying to drozzy:

I think simple argument of "--verbosity=2" will let you know which tests failed.

Yes, that will show you what failed (if you happen to notice it before it scrolls off the screen), but you don't get the failure details until the end of the run. If you are running the whole suite that may take a while. If you Ctrl-C the run after you see the first failure you still don't get the failure information so you then have to run just that test again to see if it fails in isolation...plus with this approach you have to be watching the test run like a hawk to catch the first failure. It would be more convenient if there was an option for the run to stop and print the details of the failure once the first failure is encountered. Then you can start the run and periodically check on it to see how it is going -- you don't have to watch it in detail and so long as it keeps running you know you are in good shape.

Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

That's why this is proposed as an option. It does certainly make sense when manually running the suite to check for regressions due to a code change you just made.

Changed 7 years ago by Jukka Välimaa

Attachment: failfast_1.diff added

comment:5 Changed 7 years ago by Jukka Välimaa

Cc: Jukka Välimaa added
Has patch: set

I've come across situations where something like this would have been useful. When there are 1000+ tests to run to check if anything is broken, getting some info about test failure as soon as it happens can save a lot of time.

comment:6 Changed 7 years ago by robmadole

Related post on django-developers group (there is a mention about the --failfast option).

comment:7 Changed 7 years ago by Randy Barlow

Owner: changed from nobody to Randy Barlow
Status: newassigned

w00t. This is very cool.

comment:8 Changed 7 years ago by Randy Barlow

Cc: randy@… added

Also created #12364, which is similar to this.

comment:9 Changed 7 years ago by Randy Barlow

I checked out that patch. It looks good. The unit test didn't include an init.py or a models.py, so it wouldn't run. I created a new patch that includes these files. Thanks for the patch!

comment:10 Changed 7 years ago by Randy Barlow

Cc: Randy Barlow added
Owner: changed from Randy Barlow to Karen Tracey
Status: assignednew
Triage Stage: AcceptedReady for checkin

Hey Karen, this one's ready for commit. Go failfast!

comment:11 Changed 7 years ago by Karen Tracey

Owner: changed from Karen Tracey to Randy Barlow
Triage Stage: Ready for checkinAccepted

There's a problem in adding a new keyword argument to the test runner: existing alternate test runners won't be expecting the new arg. It would be good to figure out how to add this in a way that won't cause older runners to fail.

comment:12 Changed 7 years ago by Randy Barlow

Patch needs improvement: set
Status: newassigned

if you have a function foo(arg1, arg2, ...), you can get it's args by foo.func_code.co_varnames. We can use this to find out if our arguments are accepted by the function we are about to pass them to to get around this problem.

Changed 7 years ago by Randy Barlow

Attachment: ticket_11613.patch added

Alrighty, this patch will do the trick!

comment:13 Changed 7 years ago by Randy Barlow

Owner: changed from Randy Barlow to Karen Tracey
Patch needs improvement: unset
Status: assignednew
Triage Stage: AcceptedReady for checkin

Hey Karen, now the patch only will use the failfast flag if the test runner will take it. Thanks!

comment:14 Changed 7 years ago by Karen Tracey

Resolution: fixed
Status: newclosed

(In [11843]) Fixed #11613: Added a failfast option for test running. Thanks jukvalim and Randy Barlow.

comment:15 Changed 7 years ago by Karen Tracey

(In [11846]) Added tet runner tests mistakenly left out of r11843. Refs #11613.

comment:16 Changed 5 years ago by Jacob

milestone: 1.2

Milestone 1.2 deleted

Note: See TracTickets for help on using tickets.
Back to Top