Refactor runtests.py to allow using other test runners.

[Daniel Hahler]

I think it would be very nice if pytest could be used to run Django's tests.

I've started working on this, by using a pytest plugin/conftest, which reuses parts from tests/runtests.py, which I've therefore started to make more modular (​https://github.com/django/django/pull/11289).

It is not meant to replace the current test runner (or tests), but would just allow to use pytest out of the box seamlessly, with all the additional features and plugins it provides.

Is this something that is wanted in general?

[Carlton Gibson]

Hi Daniel.

I'm very happy for some refactoring to help re-use here.
(What's counts as "Fixed" for this ticket?)

(Aside: would a test runner subclass letting us use plain asserts ever get passed the line? Any interest in writing such? Fancy pinging the DevelopersMailingList?)

[Daniel Hahler]

What's counts as "Fixed" for this ticket?

When it is good enough for the pytest plugin/integration.

would a test runner subclass letting us use plain asserts ever get passed the line?

Not sure I follow, but the pytest wrapper would not be a subclass of the Django test runner, would would rather use it, and parts from tuntests. Any interest in writing such?

pytest then would allow to use plain asserts in tests - but that is out of scope here.

[David Smith]

Hi all,

I've had a bit of a look at running the Django test suite with other test runners (pytest).

One of the issues I've come accross is there are some classes which are designed to only be inherited but start with the word 'Test'. Pytest does not know this, tries to run them and fails. A name change for these would be required. Here is an example:

​https://github.com/django/django/blob/8dabdd2cc559a66b519e2a88b64575d304b96ebe/tests/staticfiles_tests/cases.py#L95

I'm happy to investigate pytest further but can I confirm that I've understood the ticket correctly (i.e. get the test suite running with pytest) and would these type of changes be welcome. I'm sure there will be other issues, but in addition to configuring a new test running it is likely to also result in some of the existing tests need to be renamed / refactored, (subtests could be an issue, for example).

[Claude Paroz]

Maybe someone could summarize here the advantages of running Django's own tests with a different test runner?

[David Smith]

All,

It will need someone much more experienced than myself to properly articulate this, and to be credible on the topic. Carlton - I may be reading too much into this but your comments sounded interested / positive on this?

I think the benefits are subjective given current test runner works so any benefits above that are about user experience. So may be hard to make a good technical reason for the change. A few of my thoughts are:

Pros
#22449 - Pytest outputs have colour so would not need to fix this ticket

• Better output - get % progress, more information on failed tests.
• Integration with IDEs (e.g. can use pytest add-in for VSCode?)
• ability to use other pytest and pytest-addins functionality. e.g. pytest --durations=50 outputs the 50 slowest tests.
• Would pytest-xdist help speed up the tests?. (currently GSOC project on this topic)
• IF project went full pytest then could use plain asserts.

Cons

• Some complications with implementation (e.g. subtests and some tests may need to be re-written)
• If we went full pytest then it would be a dependency (Django tends to avoid doing this)
• Current test runner may be better understood than getting pytest to work (not sure how much expertise we have on the current test runner?)

Ideally I'd see this as a two step changes. 1) get it working with pytest and the current test runner, 2) consider moving to pytest fully, this may be after 1 has been working for a period of time.

Appreciate thoughts on next steps (is this a topic for the mailing list?)

[Carlton Gibson]

...but your comments sounded interested / positive on this?

I think if a refactoring is available that would allow an option here, then that would be worth investigating. unittest isn't going anywhere but pytest is where all the progress is at the moment, and if there was the option to take advantage of that, well... (end of previous sentence).

I had assumed Daniel had intended to work on the ticket... No issues that not (as yet) but I think it's more a Someday/Maybe at this point. (Updating: wouldn't object if somebody close it as needsinfo, i.e. what the refactoring might look like, instead.)

Appreciate thoughts on next steps ...

It needs a proof of concept. Here's what it would take... — Not sure that's a good ROI on time spent: it's one thing for a pytest/pytest-django maintainer to do it (they know the issues well) — it's another thing for everyone else. Maybe as a learning exercise but...

So, having got thins far, I'm going to close it as needsinfo as a Someday/Maybe:

1. Who's going to take it on? (Answer that, we can re-open, and assign.)
2. What does the patch look like? (Whether it's more than a Maybe...)

I hope that makes sense. (Do make any changes if you're not in agreement!)

Thanks for the thought and investigation David!

[Nick Pope]

Replying to David Smith:

• Some complications with implementation (e.g. subtests and some tests may need to be re-written)

Note that there is the ​pytest-subtests plugin which adds support for unittest's subTests().