Opened 10 years ago

Closed 3 years ago

Last modified 3 years ago

#24522 closed New feature (fixed)

Add test command option to run tests in a random order

Reported by: Andreas Savvides Owned by: Chris Jerdonek
Component: Testing framework Version: dev
Severity: Normal Keywords:
Cc: Ahmad A. Hussein, Adam Wróbel, Chris Jerdonek, Adam Johnson Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Similar in spirit to --reverse introduced in 1.8, the idea is to add a --random option that will randomise the test run order. Grouping by test class would be preserved when using this option.

I think the idea behind --reverse is great, so I see something like --random a way of going a step further to help with debugging test cases that are not properly isolated.

Going a step further - which might be unnecessary, so this is just a minor aside - we could even add an option that generates and runs X randomised test runs in all possible orders given a set of tests. That way, people who are really paranoid can confirm that their tests really pass and order genuinely does not matter in any case.

Change History (20)

comment:1 by Tim Graham, 10 years ago

I think we would need some way of writing the order to disk so a particular random order could be reproduced. I'm not sure it's worth it.

comment:2 by Tim Graham, 10 years ago

Resolution: wontfix
Status: newclosed

If you feel strongly about this, please raise it on the DevelopersMailingList so we can get some more opinions.

comment:3 by Dan Palmer, 5 years ago

For what it's worth, I don't think this needs writing to disk.

I propose that with --random Django generates a random seed and prints it out at the beginning of the test run. This will likely be stored in shell scrollback, or on disk/in a database if running tests in a CI environment. From there, tests could be re-run with a given seed with --random-seed=XXXXX.

comment:4 by Ahmad A. Hussein, 4 years ago

Cc: Ahmad A. Hussein added
Component: UncategorizedTesting framework
Type: UncategorizedNew feature

I agree with Dan, we could use a random seed to accomplish this. This is worth looking into to expand our test runner's functionality.

It is something already supported in pytest though.

comment:5 by Adam Wróbel, 4 years ago

Cc: Adam Wróbel added

comment:6 by Chris Jerdonek, 4 years ago

Cc: Chris Jerdonek added

comment:7 by Chris Jerdonek, 4 years ago

I just posted a message to the developers' list asking if this ticket could be reopened: https://groups.google.com/g/django-developers/c/68LZwTr2vpI/m/m3wlER5rAAAJ

comment:8 by Adam Johnson, 4 years ago

Cc: Adam Johnson added

comment:9 by Chris Jerdonek, 4 years ago

Has patch: set
Resolution: wontfix
Status: closednew

I'm reopening the ticket based on the mailing list discussion (a number of people replied in support).

I also posted a PR here: https://github.com/django/django/pull/14137

comment:10 by Mariusz Felisiak, 4 years ago

Triage Stage: UnreviewedAccepted

comment:11 by Chris Jerdonek, 4 years ago

By the way, while working on the PR, I realized that we could name the option --shuffle instead of --random. I felt --shuffle was more specific and self-descriptive, whereas --random can be used in broader contexts (e.g. altering random data in tests themselves). So if we have the opportunity to be more specific in naming, I thought we should take it.

comment:12 by Adam Johnson, 4 years ago

--shuffle sounds good to me.

comment:13 by Mariusz Felisiak, 4 years ago

Owner: changed from nobody to Chris Jerdonek
Patch needs improvement: set
Status: newassigned

comment:14 by Chris Jerdonek, 4 years ago

Patch needs improvement: unset

comment:15 by Chris Jerdonek, 4 years ago

Summary: Add --random option to test command to run test cases in random orderAdd test command option to run tests in a random order

comment:16 by Adam Johnson, 4 years ago

Patch needs improvement: set

comment:17 by Chris Jerdonek, 3 years ago

Patch needs improvement: unset

comment:18 by Mariusz Felisiak, 3 years ago

Triage Stage: AcceptedReady for checkin

comment:19 by Mariusz Felisiak <felisiak.mariusz@…>, 3 years ago

Resolution: fixed
Status: assignedclosed

In 90ba716b:

Fixed #24522 -- Added a --shuffle option to DiscoverRunner.

comment:20 by GitHub <noreply@…>, 3 years ago

In f5dccbaf:

Refs #24522 -- Fixed code comment about seeds in Shuffler.init().

Note: See TracTickets for help on using tickets.
Back to Top