Document coverage setup for parallel tests

[Mr. Glass]

Ticket #31431 requested django implement support coverage for parallel tests. It was marked as wontfix, citing #4501 where a decision was made to document how to run coverage rather than integrate it.

Unfortunately, the ​coverage documentation does not mention parallel tests at all.

Since parallel tests use subprocesses, ​special setup is needed for coverage to be accurate. I have so far been unable to figure this out, and the best information I've found is in this ​StackOverflow post.

The coverage section needs documentation covering parallel tests, and the parallel tests section should probably mention that coverage needs special setup and link to it.

I would be happy to update the documentation if someone can provide me with the working answer.

[Mariusz Felisiak]

Tentatively accepted, I'm not sure if it's doable to capture coverage data with parallel tests. You can try to ask on ​support channels.

[Adam Johnson]

Coverage.py documents ​for 'run':

Coverage.py can measure multi-threaded programs by default. If you are using more exotic concurrency, with the multiprocessing, greenlet, eventlet, or gevent libraries, then coverage.py will get very confused. Use the --concurrency switch to properly measure programs using these libraries. Give it a value of multiprocessing, thread, greenlet, eventlet, or gevent. Values other than thread require the C extension.

Plus it has a whole page on the topic: ​https://coverage.readthedocs.io/en/latest/subprocess.html

I've found --concurrency=multiprocessing works fine.

Not sure Django needs to doc anything considering it's really extensively documented there.

[Mr. Glass]

Replying to Adam Johnson:

Coverage.py documents ​for 'run':

Coverage.py can measure multi-threaded programs by default. If you are using more exotic concurrency, with the multiprocessing, greenlet, eventlet, or gevent libraries, then coverage.py will get very confused. Use the --concurrency switch to properly measure programs using these libraries. Give it a value of multiprocessing, thread, greenlet, eventlet, or gevent. Values other than thread require the C extension.

Plus it has a whole page on the topic: ​https://coverage.readthedocs.io/en/latest/subprocess.html

I've found --concurrency=multiprocessing works fine.

Not sure Django needs to doc anything considering it's really extensively documented there.

The --concurrency=multiprocessing flag didn't work for me, nor a bunch of config file options I tried. I spent way more time than allotted in my sprint trying to get this, but I could be making a simple mistake. I'll try to hop on a help channel if I can buy time.

[Adam Johnson]

I just set up a demo repo showing how to integrate coverage with django's test runner here: ​https://github.com/adamchainz/django-coverage-example

Hope that helps.

[Mariusz Felisiak]

We could improve ​docs by removing "Coverage should be run in a single process to obtain accurate statistics." and changing the current commands to the:

coverage erase
coverage run --concurrency=multiprocessing ./runtests.py --settings=test_sqlite
coverage combine && coverage html

However, this raises a warning for me:

coverage/inorout.py:530: CoverageWarning: Module django was previously imported, but not measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")

[Martin Pauly]

Pull Request: ​https://github.com/django/django/pull/15116

[Mariusz Felisiak]

​New PR

[GitHub <noreply@…>]

In 69352d85:

Fixed #33213 -- Doc'd testing code coverage in parallel and used it.

[GitHub <noreply@…>]

In 78da5ca:

Reverted "Fixed #33213 -- Doc'd testing code coverage in parallel and used it."

This reverts commit 69352d85fa8412865db9e0c7f177b333c0eac3e2.

Test coverage for async methods was no longer calculated with this
change.

[GitHub <noreply@…>]

In 070cbac:

Restored multiprocessing concurrency on coverage.py settings

• Revert "Reverted "Fixed #33213 -- Doc'd testing code coverage in parallel and used it.""

This reverts commit 78da5ca0c1f2ab3201f8f6cd629e80d805ea023d.

• Restored coverage multiprocess concurrency with threads

Investigating ​https://github.com/nedbat/coveragepy/issues/1585 revealed
that thread tracing gets disabled when passing
concurrency = multiprocessing. Adding thread restores it, and
ensures that the auser() is reported as covered since the test suite
uses AsyncToSync to execute this middleware (which spawns threads).