Coverage integration with multiprocessed Django tests.

[Ichlasul Affan]

Currently, to use Coverage.py for line coverage analysis on Django tests, we can use this command:

coverage run --omit='....' manage.py test .....
coverage report -m

This method is valid on single-processed testing. Coverage.py will analyze which lines that Python interpreter has gone through when the testing process runs. Unfortunately, on multi-processed testing, coverage analysis result becomes invalid as it marks some lines as "uncovered", given the same code and test suites.

Based on my initial observation, looks like the Coverage.py only analyze lines run by main Python interpreter process. The child processes are untouched. So, only tests run by the first process that are being analyzed. For example, I have two different test suites:

• AccountTests for account app, and
• HistoryTests for history app.

Let's say the first test process will run AccountTests. The result will only mark as "covered" on code lines in account app, and some of the models.py, urls.py, and apps.py code of history app. The latter happened because Django initialized App registry, test models, and migrations using codes in apps.py and models.py. The parent urls.py also imports urls.py inside each app.

These are subfeatures that I propose in this ticket:

1. Integration of Coverage.py into Django Test engine, so we can use coverage analysis by using: python manage.py test --check-coverage --coverage-omit='....' ....... The execution of Coverage.py is now handled by Django test using different test runner called CoverageTestRunner, that extends the ordinary DiscoverRunner.
2. Implementation of CoverageTestRunner itself, by utilizing Coverage.py's own API. This includes initialization process for coverage analysis inside the test runner.
3. Implementation of multiprocessed test runner support. It might be implemented using different init worker than the usual ParallelTestSuite, or there might be any better idea in the future. When merging test results into parent process, it will also need to merge the coverage analysis result of each processes.

I have no idea yet for a concrete Proof of Concept. The main idea is initializing coverage analysis API for each of test processes, then merge the results using OR operator. That said, a line counts as "covered" when it is executed by minimum of one test process. Any suggestions about this idea will be very much appreciated.

[Mariusz Felisiak]

Thanks for this ticket, however it was already discussed in #4501 and we decided that it would be a overkill, see comment.

[Carlton Gibson]

One bit of this I wouldn't mind seeing is capturing coverage data when --parallel is greater than 1. At the moment we can't do that. (And it's a lot slower running the whole test suite single threaded: doable, sure, but just adds friction.)

(Agree that coverage run ... isn't much of an issue.)