#11613 closed (fixed)
Add a --failfast argument to the testrunner
Reported by: | Alex Gaynor | Owned by: | Karen Tracey |
---|---|---|---|
Component: | Testing framework | Version: | 1.0 |
Severity: | Keywords: | ||
Cc: | kmike84@…, Jukka Välimaa, randy@…, Randy Barlow | Triage Stage: | Ready for checkin |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
This would basically signal the testrunner to stop and report the failure as soon as the failure occured, rather than obligating the user to wait for the test suite to finis running to see the failure. This is useful while testing for regressions.
Attachments (2)
Change History (18)
comment:1 by , 15 years ago
Cc: | added |
---|
comment:2 by , 15 years ago
Triage Stage: | Unreviewed → Accepted |
---|
follow-up: 4 comment:3 by , 15 years ago
comment:4 by , 15 years ago
Replying to drozzy:
I think simple argument of "--verbosity=2" will let you know which tests failed.
Yes, that will show you what failed (if you happen to notice it before it scrolls off the screen), but you don't get the failure details until the end of the run. If you are running the whole suite that may take a while. If you Ctrl-C the run after you see the first failure you still don't get the failure information so you then have to run just that test again to see if it fails in isolation...plus with this approach you have to be watching the test run like a hawk to catch the first failure. It would be more convenient if there was an option for the run to stop and print the details of the failure once the first failure is encountered. Then you can start the run and periodically check on it to see how it is going -- you don't have to watch it in detail and so long as it keeps running you know you are in good shape.
Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.
That's why this is proposed as an option. It does certainly make sense when manually running the suite to check for regressions due to a code change you just made.
by , 15 years ago
Attachment: | failfast_1.diff added |
---|
comment:5 by , 15 years ago
Cc: | added |
---|---|
Has patch: | set |
I've come across situations where something like this would have been useful. When there are 1000+ tests to run to check if anything is broken, getting some info about test failure as soon as it happens can save a lot of time.
comment:6 by , 15 years ago
Related post on django-developers group (there is a mention about the --failfast option).
comment:9 by , 15 years ago
I checked out that patch. It looks good. The unit test didn't include an init.py or a models.py, so it wouldn't run. I created a new patch that includes these files. Thanks for the patch!
comment:10 by , 15 years ago
Cc: | added |
---|---|
Owner: | changed from | to
Status: | assigned → new |
Triage Stage: | Accepted → Ready for checkin |
Hey Karen, this one's ready for commit. Go failfast!
comment:11 by , 15 years ago
Owner: | changed from | to
---|---|
Triage Stage: | Ready for checkin → Accepted |
There's a problem in adding a new keyword argument to the test runner: existing alternate test runners won't be expecting the new arg. It would be good to figure out how to add this in a way that won't cause older runners to fail.
comment:12 by , 15 years ago
Patch needs improvement: | set |
---|---|
Status: | new → assigned |
if you have a function foo(arg1, arg2, ...), you can get it's args by foo.func_code.co_varnames. We can use this to find out if our arguments are accepted by the function we are about to pass them to to get around this problem.
comment:13 by , 15 years ago
Owner: | changed from | to
---|---|
Patch needs improvement: | unset |
Status: | assigned → new |
Triage Stage: | Accepted → Ready for checkin |
Hey Karen, now the patch only will use the failfast flag if the test runner will take it. Thanks!
comment:14 by , 15 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
I think simple argument of "--verbosity=2" will let you know which tests failed.
Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.