Show the duration of individual tests

[Paolo Melchiorre]

I propose to add the possibility to show the duration of individual tests in the final report by ordering them from slowest to fastest and filtering only the slowest tests with a threshold.

Example:

$ python -m manage test --durations=10
Found 92 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
Creating test database for alias 'default'...
..........
----------------------------------------------------------------------
Ran 92 tests in 1.297s

OK
Destroying test database for alias 'default'...

Slowest 10 test duration durations:
0.3597s test_detail_view_with_a_future_poll (polls.tests.PollIndexDetailTests)
0.0284s test_detail_view_with_a_past_poll (polls.tests.PollIndexDetailTests)
0.0068s test_index_view_with_a_future_poll (polls.tests.PollViewTests)
0.0047s test_index_view_with_a_past_poll (polls.tests.PollViewTests)
0.0045s test_index_view_with_two_past_polls (polls.tests.PollViewTests)
0.0041s test_index_view_with_future_poll_and_past_poll (polls.tests.PollViewTests)
0.0036s test_index_view_with_no_polls (polls.tests.PollViewTests)
0.0003s test_was_published_recently_with_future_poll (polls.tests.PollMethodTests)
0.0002s test_was_published_recently_with_recent_poll (polls.tests.PollMethodTests)
0.0002s test_was_published_recently_with_old_poll (polls.tests.PollMethodTests)

pytest has a very similar option that helps you find slower tests that you want to optimize their timing.
​https://pytest.org/en/latest/how-to/usage.html#durations

There's an external package django-slow-tests that adds a similar functionality, but the code is very old and with very little test coverage, furthermore the last release was almost 4 years ago and the officially supported versions of Django are very old.
​https://github.com/realpython/django-slow-tests

I propose to add this feature directly into Django tests to help users optimize their tests and also to help Django developers optimize the test of Dajngo itself.

[Carlton Gibson]

I think this would be great. (I'm sure Adam mentioned similar once upon a time… 🤔)

I could do with the same for Django's own test suite — there's one test that I notice sitting there, which would be good to get a bead on 🙂

[Paolo Melchiorre]

Replying to Carlton Gibson:

I think this would be great. (I'm sure Adam mentioned similar once upon a time… 🤔)

Sure, Adam talked about the two alternatives in his book.

Ever since I read this, I started wondering if it might not have been a good idea to have this function directly in Django.

I could do with the same for Django's own test suite — there's one test that I notice sitting there, which would be good to get a bead on 🙂

By often running Django tests locally, lately I too have noticed that for a particular test the tests remain pending for a few seconds and it would not hurt to find out which one it is and optimize it.

[Mariusz Felisiak]

As far as I'm aware, we declined to add --durations in this ​PR.

[Carlton Gibson]

I think last comment there was mine here ​https://github.com/django/django/pull/13224#issuecomment-664913219 — pro durations but handling it separately, so here maybe :)

[Adam Johnson]

django-rich has an open PR from David Smith to add test timing functionality: ​https://github.com/adamchainz/django-rich/pull/30

It’s stalled because of sorting out parallel testing, and trying to also time setUp / tearDown / setUpClass / tearDownClass.

[Paolo Melchiorre]

Seems to me like a very useful feature to have right in Django tests.

It would be very useful to be able to use this feature also to optimize Django's tests themselves.

It could be a feature that a tool like Django Rich could add color to?

[Adam Johnson]

Yes, though the initial idea was to implement it first in a third party package before proposing it for Django itself.

[Paolo Melchiorre]

I think it's a good idea.

At the same time I thought it would be interesting to take advantage of this functionality to discover w and slow tests of Django itself, and I don't think it can be done using an external package.

[David Smith]

Python 3.12 will add a --durations flag.

​https://github.com/python/cpython/pull/12271

I am leaning towards saying thats enough to call this wontfix.

While it's likely that a bit more investigation is needed to see if it covers SetUp / TearDown I wonder if its reasonable to say let's implement that additional functionality in a 3rd party package first?

[Adam Johnson]

That sounds good. Even with the 3.12 support, Django will still need to forward the option through to unittest.

[Adam Johnson]

Oh, and the unittest flag will not work with parallel testing - there will need to be some work in Django to merge the test times across processes.

For future reference David opened a PR in django-rich for timing: ​https://github.com/adamchainz/django-rich/pull/30 . It stalled due to my prefectionism trying to time all setups and teardowns...

[Sarah Boyce]

I think this also will close this ticket: https://code.djangoproject.com/ticket/31827

[Mariusz Felisiak]

Replying to Sarah Boyce:

I think this also will close this ticket: https://code.djangoproject.com/ticket/31827

Probably yes, let's wait for the final release first.

[Mariusz Felisiak]

#31827 was a duplicate for showing N slowest tests.

[Natalia Bidart]

Setting the flag "patch needs improvement" to remove this work from the review queue (and also because the branch will need a rebase and conflict fixes before landing).

[Mariusz Felisiak <felisiak.mariusz@…>]

In 74b50741:

Fixed #34210 -- Added unittest's durations option to the test runner.