Running tests outside the test runner needs clarification

[Stanislas Guerra]

Hi devs,

Some context first

For one of my pluggable application I have two different test suites, one of them being a Django dummy project.
In order to have a correct Coverage report, I run both of them in a runtests.py script.

​The Documentation is a bit confusing and I think a working snippet could be useful here.

For example, it is not required to execute setup_test_environment() nor django.test.runner.DiscoverRunner.setup_databases() if you execute DiscoverRunner.run_tests() which is what you want in the end.

The following snippet could be useful to the newcomers don't you think ?

import os
import sys
import django
from django.test.runner import DiscoverRunner

sys.path.insert(0, '../..')  # path to the development sources.
sys.path.insert(0, 'test_project')  # path the the test project.
os.environ['DJANGO_SETTINGS_MODULE'] = 'test_project.settings'
os.environ['DJANGO_LIVE_TEST_SERVER_ADDRESS'] = '127.0.0.1:8082'

django.setup()
runner = DiscoverRunner()
runner.run_tests(['my_app'])

[Tim Graham]

Yes, something like this seems like a good idea. We shouldn't suggest using sys.path though, per #19941 (comment 9 and later). Would you like to submit a patch? I'll be happy to review it.

[Stanislas Guerra]

Replying to timgraham:

Yes, something like this seems like a good idea. We shouldn't suggest using sys.path though, per #19941 (comment 9 and later).

In my case - AFAIR - using PYTHONPATH does not works because it came after installed packages in the resolution order (because I want my test project to run against the source located in ../.. and not against the sources I may have system-wide installed.
This is only true for the first sys.path.insert(0, '../../'), the test_project path can be put in PYTHONPATH.
In my case, I prefered not to mix both.

Anyway, the first insert is very specific and can be removed and the second converted in a PYTHONPATH expression.

Would you like to submit a patch? I'll be happy to review it.

With pleasure.

[Stanislas Guerra]

PR: ​https://github.com/django/django/pull/3485

[Preston Timmons]

This example is okay. I have a couple thoughts, though:

I assume the most common case for this is creating a test runner for reusable apps. In that case, messing with PYTHONPATH is often unnecessary since you can either pip install --editable or python setup.py develop to put your packages on the path. Is it worth mentioning this?

The example hard-codes the app label in runner.run_tests(['my_app']). Even with reusable apps it's nice to have the option of passing a path to a test case, like ./runtests.py myapp.tests.SomeCase.test. Should the example take that into consideration?

I usually do something like:

runner.runtests(sys.argv[1:] or ["myapp"])

[Stanislas Guerra]

Replying to prestontimmons:

This example is okay. I have a couple thoughts, though:

I assume the most common case for this is creating a test runner for reusable apps. In that case, messing with PYTHONPATH is often unnecessary since you can either pip install --editable or python setup.py develop to put your packages on the path. Is it worth mentioning this?

Hi Preston,

The PYTHONPATH in the PR snippet is about resolving DJANGO_SETTINGS_MODULE by adding the path of the test project, not the reusable app dev sources path (which is assumed). But maybe this is not clear enough ?

The example hard-codes the app label in runner.run_tests(['my_app']). Even with reusable apps it's nice to have the option of passing a path to a test case, like ./runtests.py myapp.tests.SomeCase.test. Should the example take that into consideration?

I usually do something like:

runner.runtests(sys.argv[1:] or ["myapp"])

Good point.
I wanted the snippet as simple as possible (how to convert a manage.py test shell invocation into Python code). In that regard, I would replace sys.argv[1:] or ['myapp'] by sys.argv[1:] and update the runtests.py invocation example with some optional arguments à la Django ([path.to.modulename|path.to.modulename.TestCase|path.to.modulename.TestCase.test_method]).
What do you think ?

[Preston Timmons]

I think there's merit to keeping your example minimal.

I just started imagining a best-practice type section of the docs for running tests in reusable apps. It would cover things like:

• Using the test runner in a script
• Managing python path
• Adding test-only models
• Configuring settings manually when including an example project is overkill
• etc.

This assumes there's a relative consensus of what "best practice" means.

I don't think that's in scope of this ticket, though.

[Stanislas Guerra]

This came to my mind too: a more elaborate example in ​advanced tutorial documentation (How to write reusable apps).
The Django ​runtests.py script and the adopted structure of the test packages looks like the obvious starting point to me.

But as you said, this is for another ticket!

[Stanislas Guerra]

​PR updated with a more elaborated example as suggested by @carljm too.

[Stanislas Guerra]

​PR updated to get something minimalistic. Thanks @carljm for putting me on that path!

[Carl Meyer <carl@…>]

In 06726965c3e53e9a6b87e1532951a93d5f94f426:

Fixed #23778 -- Added a doc section on using the Django runner for reusable apps.

[Carl Meyer <carl@…>]

In 4ea39230fc2c77cc091ebbbdb3d445920b299746:

[1.7.x] Fixed #23778 -- Added a doc section on using the Django runner for reusable apps.

Backport of 06726965c3e53e9a6b87e1532951a93d5f94f426 from master.