Opened 9 years ago

Last modified 5 weeks ago

#11593 assigned New feature

Incomplete support for app-level testing

Reported by: Masklinn Owned by: Raphael Kimmig
Component: Testing framework Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Django and its community have a pretty strong focus on the concept of reusable applications: applications as pluggable behavior blocks, independent from one another and from projects. And the test framework documentation and setup emphasize that: the documentation page is called Testing Django applications and the test themselves are saved at the application level not at the project level.

Yet Django provides no provision whatsoever for running tests from an application: a project (or at least a settings and a urls file) is required, and barring fairly weird setups (global settings and urls modules) there is no way to create a new app, write a test and run it just like that. Which means applications aren't as self-contained as they could be, and can't easily be tested "in a void" (independently of everything else).

Change History (15)

comment:1 Changed 9 years ago by Russell Keith-Magee

Triage Stage: UnreviewedAccepted

I concur 100%. The distinction between integration testing of new apps, and system testing of a specific app is a persistent problem, and the cause of more than one ticket in the Django system (e.g., #10976, #10755, and others). This is a problem for which we should have a solid answer.

comment:2 in reply to:  1 ; Changed 9 years ago by Manoj Govindan <egmanoj@…>

Replying to russellm:

I concur 100%. The distinction between integration testing of new apps, and system testing of a specific app is a persistent problem, and the cause of more than one ticket in the Django system (e.g., #10976, #10755, and others). This is a problem for which we should have a solid answer.


Can I make a wish while we are on the topic? ;) Django allows me to write test suites (unittest.TestSuite objects) but won't allow me to run them. Adding one more level of granularity would help users to run any of Application/Test Suite/Test Class/Test Method tests. This would also make django's test running mechanism more similar to unittest's.

If you think this should be a separate ticket I can add one.

comment:3 Changed 9 years ago by Manoj Govindan <egmanoj@…>

Cc: egmanoj@… added

comment:4 in reply to:  2 ; Changed 9 years ago by Russell Keith-Magee

Replying to Manoj Govindan <egmanoj@gmail.com>:

Replying to russellm:
Can I make a wish while we are on the topic? ;) Django allows me to write test suites (unittest.TestSuite objects) but won't allow me to run them. Adding one more level of granularity would help users to run any of Application/Test Suite/Test Class/Test Method tests. This would also make django's test running mechanism more similar to unittest's.

What you are proposing isn't a stylistic thing or a particular way that the original proposal could be interpreted - it's a completely new idea. If you have a separate feature request, open a separate ticket.

comment:5 in reply to:  4 Changed 9 years ago by Manoj Govindan <egmanoj@…>

What you are proposing isn't a stylistic thing or a particular way that the original proposal could be interpreted - it's a completely new idea. If you have a separate feature request, open a separate ticket.


Done.

comment:6 Changed 8 years ago by Julien Phalip

Severity: Normal
Type: New feature

comment:7 Changed 7 years ago by Aymeric Augustin

UI/UX: unset

Change UI/UX from NULL to False.

comment:8 Changed 7 years ago by Aymeric Augustin

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:9 Changed 6 weeks ago by Raphael Kimmig

Owner: changed from nobody to Raphael Kimmig
Status: newassigned

This ticket is really old, but I'd still love to see some progress here.
I've been thinking about this a bit lately and below are my thoughts so far.
We probably don't need to write a lot of code here, but rather add some
documentation to the advanced tutorial as well as the advanced testing
documentation.

From what I can see there are a few common things that should be documented.

  1. How to run tests in a stand alone application (this requires settings).
  2. How to manage settings for running tests in a stand alone application.
  3. How to set up views/urls/middleware ... for tests.
  4. How to add models specifically for tests inside applications.

For 1, 2 and 3. there are several solutions, to get started I'd probably describe
how to use django-admin test to run the tests with a custom test_settings.py
file.

For 4. there is a simple solution if the app doesn't provide any models itself.
In that case one can simply disable migrations for the app in the test settings.
Ideally I'd like to try implementing a @test_model decorator as suggested in
https://code.djangoproject.com/ticket/7835 which would also solve the general
case.

I'd love to have the documentation for reusable apps contain (directly, or via links
to the testing docs) all the information required to not only create an app, but also
make sure it has working tests.

I'll try to expand the relevant docs over the next few days. If anyone has some
input on the matter I'd be glad to hear it.

Did I miss any obvious work in that area? Maybe even somewhere in the documentation?
Do you have any suggestions on how specific aspects should be solved?

/e
After looking some more I've found the section in the advanced testing docs that covers
most of this already :-)

https://docs.djangoproject.com/en/2.1/topics/testing/advanced/#using-the-django-test-runner-to-test-reusable-applications

I think the one thing that is really missing is some mention of that in the reusable apps introduction.
Also the part in advanced testing that covers how to use models only for tests could probably be worded
a bit more clearly.

Last edited 6 weeks ago by Raphael Kimmig (previous) (diff)

comment:10 Changed 5 weeks ago by Raphael Kimmig

I've added documentation to the resuable apps intro. PR

One possible way to improve this would be to add a --reusable flag to the test management command so that one could run the tests with django-admin test --reusable or something along those lines instead of copy & pasting a runtests.py file. Any thoughts?

Last edited 5 weeks ago by Simon Charette (previous) (diff)

comment:11 Changed 5 weeks ago by Raphael Kimmig

Has patch: set

comment:12 Changed 5 weeks ago by Simon Charette

Is there a reason why python -m django test --settings=tests.settings tests (given tests.test_settings is renamed to settings) wouldn't work?

That's the pattern I use for all the third party apps I maintain and it seems to be working just fine.

e.g.

I'm not sure I'm following why a --reusable flag is required.

comment:13 Changed 5 weeks ago by Raphael Kimmig

No good reason at all, that looks much nicer actually :-)

I'll update the PR and I'll change the advanced testing part about reusable applications as well,
unless you see a good reason to suggest using a runtests.py file there.

comment:14 Changed 5 weeks ago by Simon Charette

I don't see a reason for shipping a runtests.py file with minimal configuration. The one we use for Django's testsuite happens to exist because we need to perform quite a lot of tweaks before running the tests which isn't the case for most Django third party apps.

comment:15 Changed 5 weeks ago by Tim Graham

Cc: egmanoj@… removed
Note: See TracTickets for help on using tickets.
Back to Top