Opened 3 years ago
Closed 20 months ago
#33213 closed New feature (fixed)
Document coverage setup for parallel tests
Reported by: | Mr. Glass | Owned by: | Paolo Melchiorre |
---|---|---|---|
Component: | Documentation | Version: | |
Severity: | Normal | Keywords: | |
Cc: | Carlton Gibson | Triage Stage: | Accepted |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Ticket #31431 requested django implement support coverage for parallel tests. It was marked as wontfix, citing #4501 where a decision was made to document how to run coverage rather than integrate it.
Unfortunately, the coverage documentation does not mention parallel tests at all.
Since parallel tests use subprocesses, special setup is needed for coverage to be accurate. I have so far been unable to figure this out, and the best information I've found is in this StackOverflow post.
The coverage section needs documentation covering parallel tests, and the parallel tests section should probably mention that coverage needs special setup and link to it.
I would be happy to update the documentation if someone can provide me with the working answer.
Change History (15)
comment:1 by , 3 years ago
Component: | Uncategorized → Documentation |
---|---|
Type: | Uncategorized → New feature |
comment:2 by , 3 years ago
Cc: | added |
---|---|
Triage Stage: | Unreviewed → Accepted |
follow-up: 4 comment:3 by , 3 years ago
Coverage.py documents for 'run':
Coverage.py can measure multi-threaded programs by default. If you are using more exotic concurrency, with the multiprocessing, greenlet, eventlet, or gevent libraries, then coverage.py will get very confused. Use the --concurrency switch to properly measure programs using these libraries. Give it a value of multiprocessing, thread, greenlet, eventlet, or gevent. Values other than thread require the C extension.
Plus it has a whole page on the topic: https://coverage.readthedocs.io/en/latest/subprocess.html
I've found --concurrency=multiprocessing
works fine.
Not sure Django needs to doc anything considering it's really extensively documented there.
comment:4 by , 3 years ago
Replying to Adam Johnson:
Coverage.py documents for 'run':
Coverage.py can measure multi-threaded programs by default. If you are using more exotic concurrency, with the multiprocessing, greenlet, eventlet, or gevent libraries, then coverage.py will get very confused. Use the --concurrency switch to properly measure programs using these libraries. Give it a value of multiprocessing, thread, greenlet, eventlet, or gevent. Values other than thread require the C extension.
Plus it has a whole page on the topic: https://coverage.readthedocs.io/en/latest/subprocess.html
I've found
--concurrency=multiprocessing
works fine.
Not sure Django needs to doc anything considering it's really extensively documented there.
The --concurrency=multiprocessing flag didn't work for me, nor a bunch of config file options I tried. I spent way more time than allotted in my sprint trying to get this, but I could be making a simple mistake. I'll try to hop on a help channel if I can buy time.
comment:5 by , 3 years ago
I just set up a demo repo showing how to integrate coverage with django's test runner here: https://github.com/adamchainz/django-coverage-example
Hope that helps.
comment:6 by , 3 years ago
We could improve docs by removing "Coverage should be run in a single process to obtain accurate statistics." and changing the current commands to the:
coverage erase coverage run --concurrency=multiprocessing ./runtests.py --settings=test_sqlite coverage combine && coverage html
However, this raises a warning for me:
coverage/inorout.py:530: CoverageWarning: Module django was previously imported, but not measured (module-not-measured) self.warn(msg, slug="module-not-measured")
comment:7 by , 3 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:8 by , 3 years ago
Has patch: | set |
---|
Pull Request: https://github.com/django/django/pull/15116
comment:9 by , 3 years ago
Patch needs improvement: | set |
---|
comment:10 by , 23 months ago
Owner: | changed from | to
---|---|
Patch needs improvement: | unset |
Triage Stage: | Accepted → Ready for checkin |
comment:13 by , 22 months ago
Resolution: | fixed |
---|---|
Status: | closed → new |
Triage Stage: | Ready for checkin → Accepted |
comment:14 by , 22 months ago
Has patch: | unset |
---|
Tentatively accepted, I'm not sure if it's doable to capture coverage data with parallel tests. You can try to ask on support channels.