#24522 closed New feature (fixed)
Add test command option to run tests in a random order
Reported by: | Andreas Savvides | Owned by: | Chris Jerdonek |
---|---|---|---|
Component: | Testing framework | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Ahmad A. Hussein, Adam Wróbel, Chris Jerdonek, Adam Johnson | Triage Stage: | Ready for checkin |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Similar in spirit to --reverse introduced in 1.8, the idea is to add a --random option that will randomise the test run order. Grouping by test class would be preserved when using this option.
I think the idea behind --reverse is great, so I see something like --random a way of going a step further to help with debugging test cases that are not properly isolated.
Going a step further - which might be unnecessary, so this is just a minor aside - we could even add an option that generates and runs X randomised test runs in all possible orders given a set of tests. That way, people who are really paranoid can confirm that their tests really pass and order genuinely does not matter in any case.
Change History (20)
comment:1 by , 10 years ago
comment:2 by , 10 years ago
Resolution: | → wontfix |
---|---|
Status: | new → closed |
If you feel strongly about this, please raise it on the DevelopersMailingList so we can get some more opinions.
comment:3 by , 5 years ago
For what it's worth, I don't think this needs writing to disk.
I propose that with --random
Django generates a random seed and prints it out at the beginning of the test run. This will likely be stored in shell scrollback, or on disk/in a database if running tests in a CI environment. From there, tests could be re-run with a given seed with --random-seed=XXXXX
.
comment:4 by , 4 years ago
Cc: | added |
---|---|
Component: | Uncategorized → Testing framework |
Type: | Uncategorized → New feature |
I agree with Dan, we could use a random seed to accomplish this. This is worth looking into to expand our test runner's functionality.
It is something already supported in pytest though.
comment:5 by , 4 years ago
Cc: | added |
---|
comment:6 by , 4 years ago
Cc: | added |
---|
comment:7 by , 4 years ago
I just posted a message to the developers' list asking if this ticket could be reopened: https://groups.google.com/g/django-developers/c/68LZwTr2vpI/m/m3wlER5rAAAJ
comment:8 by , 4 years ago
Cc: | added |
---|
comment:9 by , 4 years ago
Has patch: | set |
---|---|
Resolution: | wontfix |
Status: | closed → new |
I'm reopening the ticket based on the mailing list discussion (a number of people replied in support).
I also posted a PR here: https://github.com/django/django/pull/14137
comment:10 by , 4 years ago
Triage Stage: | Unreviewed → Accepted |
---|
comment:11 by , 4 years ago
By the way, while working on the PR, I realized that we could name the option --shuffle
instead of --random
. I felt --shuffle
was more specific and self-descriptive, whereas --random
can be used in broader contexts (e.g. altering random data in tests themselves). So if we have the opportunity to be more specific in naming, I thought we should take it.
comment:13 by , 4 years ago
Owner: | changed from | to
---|---|
Patch needs improvement: | set |
Status: | new → assigned |
comment:14 by , 4 years ago
Patch needs improvement: | unset |
---|
comment:15 by , 4 years ago
Summary: | Add --random option to test command to run test cases in random order → Add test command option to run tests in a random order |
---|
comment:16 by , 4 years ago
Patch needs improvement: | set |
---|
comment:17 by , 3 years ago
Patch needs improvement: | unset |
---|
comment:18 by , 3 years ago
Triage Stage: | Accepted → Ready for checkin |
---|
I think we would need some way of writing the order to disk so a particular random order could be reproduced. I'm not sure it's worth it.