Opened 7 years ago

Closed 5 months ago

Last modified 5 months ago

#24522 closed New feature (fixed)

Add test command option to run tests in a random order

Reported by: Andreas Savvides Owned by: Chris Jerdonek
Component: Testing framework Version: dev
Severity: Normal Keywords:
Cc: Ahmad A. Hussein, Adam Wróbel, Chris Jerdonek, Adam Johnson Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Similar in spirit to --reverse introduced in 1.8, the idea is to add a --random option that will randomise the test run order. Grouping by test class would be preserved when using this option.

I think the idea behind --reverse is great, so I see something like --random a way of going a step further to help with debugging test cases that are not properly isolated.

Going a step further - which might be unnecessary, so this is just a minor aside - we could even add an option that generates and runs X randomised test runs in all possible orders given a set of tests. That way, people who are really paranoid can confirm that their tests really pass and order genuinely does not matter in any case.

Change History (20)

comment:1 Changed 7 years ago by Tim Graham

I think we would need some way of writing the order to disk so a particular random order could be reproduced. I'm not sure it's worth it.

comment:2 Changed 7 years ago by Tim Graham

Resolution: wontfix
Status: newclosed

If you feel strongly about this, please raise it on the DevelopersMailingList so we can get some more opinions.

comment:3 Changed 2 years ago by Dan Palmer

For what it's worth, I don't think this needs writing to disk.

I propose that with --random Django generates a random seed and prints it out at the beginning of the test run. This will likely be stored in shell scrollback, or on disk/in a database if running tests in a CI environment. From there, tests could be re-run with a given seed with --random-seed=XXXXX.

comment:4 Changed 17 months ago by Ahmad A. Hussein

Cc: Ahmad A. Hussein added
Component: UncategorizedTesting framework
Type: UncategorizedNew feature

I agree with Dan, we could use a random seed to accomplish this. This is worth looking into to expand our test runner's functionality.

It is something already supported in pytest though.

comment:5 Changed 10 months ago by Adam Wróbel

Cc: Adam Wróbel added

comment:6 Changed 9 months ago by Chris Jerdonek

Cc: Chris Jerdonek added

comment:7 Changed 9 months ago by Chris Jerdonek

I just posted a message to the developers' list asking if this ticket could be reopened: https://groups.google.com/g/django-developers/c/68LZwTr2vpI/m/m3wlER5rAAAJ

comment:8 Changed 9 months ago by Adam Johnson

Cc: Adam Johnson added

comment:9 Changed 9 months ago by Chris Jerdonek

Has patch: set
Resolution: wontfix
Status: closednew

I'm reopening the ticket based on the mailing list discussion (a number of people replied in support).

I also posted a PR here: https://github.com/django/django/pull/14137

comment:10 Changed 9 months ago by Mariusz Felisiak

Triage Stage: UnreviewedAccepted

comment:11 Changed 9 months ago by Chris Jerdonek

By the way, while working on the PR, I realized that we could name the option --shuffle instead of --random. I felt --shuffle was more specific and self-descriptive, whereas --random can be used in broader contexts (e.g. altering random data in tests themselves). So if we have the opportunity to be more specific in naming, I thought we should take it.

comment:12 Changed 9 months ago by Adam Johnson

--shuffle sounds good to me.

comment:13 Changed 7 months ago by Mariusz Felisiak

Owner: changed from nobody to Chris Jerdonek
Patch needs improvement: set
Status: newassigned

comment:14 Changed 7 months ago by Chris Jerdonek

Patch needs improvement: unset

comment:15 Changed 7 months ago by Chris Jerdonek

Summary: Add --random option to test command to run test cases in random orderAdd test command option to run tests in a random order

comment:16 Changed 7 months ago by Adam Johnson

Patch needs improvement: set

comment:17 Changed 6 months ago by Chris Jerdonek

Patch needs improvement: unset

comment:18 Changed 5 months ago by Mariusz Felisiak

Triage Stage: AcceptedReady for checkin

comment:19 Changed 5 months ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In 90ba716b:

Fixed #24522 -- Added a --shuffle option to DiscoverRunner.

comment:20 Changed 5 months ago by GitHub <noreply@…>

In f5dccbaf:

Refs #24522 -- Fixed code comment about seeds in Shuffler.init().

Note: See TracTickets for help on using tickets.
Back to Top