Automate test coverage check on pull requests

[Jacob Walls]

We could automate a significant part of checking test coverage on pull requests. Instead of using a cloud service, there's an implementation using diff-cover in a PR.

​PR

​Forum post

[Simon Charette]

I think the last time this was attempted the efforts stalled due the difficulty of combining coverage reports from different suite runs. Some part of the code is only covered by tests run on Postgres or a particular Python version for example so there needs to be a coordinated job that collects all of the .coverage artifacts and ​then combine them otherwise the resulting coverage report will be lacking or improperly reporting that some areas are not covered (e.g. if we only use the SQLite test run and Postgres only changes are introduced).

[Jacob Walls]

(The linked implementation just treats that complexity as outside of MVP and makes this caveat explicit in the posted comment.)

[Jacob Walls <jacobtylerwalls@…>]

In a89183e6:

Fixed #36620 -- Added coverage workflow to summarize coverage in pull requests.

Part of GSoC 2025. Thanks Lily for mentorship, and Sarah Boyce and
Jacob Walls for reviews.

[Jacob Walls <jacobtylerwalls@…>]

In e4c4a17:

Reverted "Fixed #36620 -- Added coverage workflow to summarize coverage in pull requests."

This reverts commit a89183e63844a937aacd3ddb73c4952ef869d2cc.

[Jacob Walls]

Revision is being progressed here: ​https://github.com/django/django/pull/20302

[GitHub <noreply@…>]

In 26b0e2bb:

Fixed #36620 -- Fixed workflow to summarize coverage in PRs.

Follow-up to a89183e63844a937aacd3ddb73c4952ef869d2cc, which was
reverted in e4c4a178aa642f8493b7ae2c0ad58527af51f67e because a change
to the workflow trigger resulted in the PR branch not being checked out.

We used this opportunity to reimplement the coverage tracing and coverage
commenting in a two-workflow pattern with more granular permissions.

To reduce duplicative workflows, we removed the existing python test workflow
on PRs, at least until we run more distinct configurations on GitHub actions. The
run with coverage tracing enabled is sufficient for now. The existing workflow still
runs on pushes to main. We can revisit when adding more test configurations.

[GitHub <noreply@…>]

In 1dfd5aa:

Refs #36620 -- Removed PR number null check from coverage_comment workflow.

[GitHub <noreply@…>]

In fd4c5fa3:

Refs #36620 -- Fixed PR number extraction in coverage_comment workflow.

Passing the PR number as an artifact is more reliable in cross-fork workflows.

[Jacob Walls <jacobtylerwalls@…>]

In e726254a:

Refs #36620 -- Added contributor documentation for code coverage reports.

This was included in the original reverted patch:
a89183e63844a937aacd3ddb73c4952ef869d2cc
Follow-up to 26b0e2bb92caf2d16cabe455792350f20d6f42ca.

[Jacob Walls <jacobtylerwalls@…>]

In 2e7133c:

[6.0.x] Refs #36620 -- Added contributor documentation for code coverage reports.

This was included in the original reverted patch:
a89183e63844a937aacd3ddb73c4952ef869d2cc
Follow-up to 26b0e2bb92caf2d16cabe455792350f20d6f42ca.

Backport of e726254a380f2a35a2fcf71143e96cb5987d8102 from main.

[Jacob Walls <jacobtylerwalls@…>]

In 2351c1b1:

Refs #36620 -- Ran coverage tests workflow on forks.

We can continue to limit the coverage comment workflow to django/django,
but now that this workflow is the main python test workflow, it should
run on forks by default. The other tests workflow (currently running
only JavaScript tests) may start running python tests again once we flesh
out the matrix, but since it was duplicating the coverage tests configuration,
we temporarily removed it.

Follow-up to 26b0e2bb92caf2d16cabe455792350f20d6f42ca.