Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#31431 closed New feature (wontfix)

Coverage integration with multiprocessed Django tests.

Reported by: Ichlasul Affan Owned by: Ichlasul Affan
Component: Testing framework Version: dev
Severity: Normal Keywords: coverage; parallel testing
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Currently, to use Coverage.py for line coverage analysis on Django tests, we can use this command:

coverage run --omit='....' manage.py test .....
coverage report -m

This method is valid on single-processed testing. Coverage.py will analyze which lines that Python interpreter has gone through when the testing process runs. Unfortunately, on multi-processed testing, coverage analysis result becomes invalid as it marks some lines as "uncovered", given the same code and test suites.

Based on my initial observation, looks like the Coverage.py only analyze lines run by main Python interpreter process. The child processes are untouched. So, only tests run by the first process that are being analyzed. For example, I have two different test suites:

  • AccountTests for account app, and
  • HistoryTests for history app.

Let's say the first test process will run AccountTests. The result will only mark as "covered" on code lines in account app, and some of the models.py, urls.py, and apps.py code of history app. The latter happened because Django initialized App registry, test models, and migrations using codes in apps.py and models.py. The parent urls.py also imports urls.py inside each app.

These are subfeatures that I propose in this ticket:

  1. Integration of Coverage.py into Django Test engine, so we can use coverage analysis by using: python manage.py test --check-coverage --coverage-omit='....' ....... The execution of Coverage.py is now handled by Django test using different test runner called CoverageTestRunner, that extends the ordinary DiscoverRunner.
  2. Implementation of CoverageTestRunner itself, by utilizing Coverage.py's own API. This includes initialization process for coverage analysis inside the test runner.
  3. Implementation of multiprocessed test runner support. It might be implemented using different init worker than the usual ParallelTestSuite, or there might be any better idea in the future. When merging test results into parent process, it will also need to merge the coverage analysis result of each processes.

I have no idea yet for a concrete Proof of Concept. The main idea is initializing coverage analysis API for each of test processes, then merge the results using OR operator. That said, a line counts as "covered" when it is executed by minimum of one test process. Any suggestions about this idea will be very much appreciated.

Change History (3)

comment:1 by Ichlasul Affan, 4 years ago

Owner: changed from nobody to Ichlasul Affan
Status: newassigned

comment:2 by Mariusz Felisiak, 4 years ago

Resolution: wontfix
Status: assignedclosed
Summary: Better Coverage Analysis integration with multiprocessed Django Test (ParallelTestSuite)Coverage integration with multiprocessed Django tests.

Thanks for this ticket, however it was already discussed in #4501 and we decided that it would be a overkill, see comment.

comment:3 by Carlton Gibson, 4 years ago

One bit of this I wouldn't mind seeing is capturing coverage data when --parallel is greater than 1. At the moment we can't do that. (And it's a lot slower running the whole test suite single threaded: doable, sure, but just adds friction.)

(Agree that coverage run ... isn't much of an issue.)

Note: See TracTickets for help on using tickets.
Back to Top