Speed up post-commit callback dispatch

[harshit gupta]

Summary:
Speed up post-commit callback dispatch

Description:

BaseDatabaseWrapper.run_and_clear_commit_hooks() drains its detached callback list with pop(0). Each removal shifts the remaining entries, so queue-management overhead is O(N²) in the number of callbacks.

This can matter for bulk work that registers a post-commit callback per item, such as deferred cache updates. Callback work itself may dominate in applications that make network or task-broker calls; those costs are not part of the queue-complexity claim.

A local patch keeps the existing list-detachment step and visits each entry once, clearing its slot before calling it. This makes queue-management overhead O(N), without allocating another queue. Clearing the slot prevents completed callbacks from being retained by the detached list or a saved traceback from a later callback failure.

The existing robust/nonrobust handling stays unchanged. The list allocation still lasts until the frame is released, and failing or unrun callbacks can still be retained by a saved traceback. An external alias to this private list would see None slots instead of a shrinking list, so private-list inspection and direct mutation are not claimed to be identical.

Reproduction and measurements:

Checked against d430084791a61b31cecec132bce3dca19194f7f2 on Python 3.14.5, Windows, SQLite.

The runner compiles the original method from that commit in memory and compares it with the changed method. Its benchmarks include transaction begin/commit, callback registration and callbacks updating an in-memory dictionary. Two workloads give callbacks separately captured bytearrays to check retained state. These are controlled workloads, not production traffic, model writes or broker calls.

There are 48 rows: 12 workloads measured separately on each of four CPU pairs. Each row has five paired timing samples per implementation and five separate tracemalloc measurements. Only two physical cores are allowed at a time, with one process and rotating CPU pairs. All rows, including slower cases, are retained. Different CPU pairs are not pooled. Peaks describe newly traced Python allocations, not total process RAM.

Verification:

• The same 195 focused tests ran on the original and changed methods, with no failures and the same 10 SQLite skips.
• Five regression tests cover exceptions, execution ordering and callback lifetimes, including retained tracebacks and error logs.
• 468 before/after comparisons matched callback order, error/log outcomes, queue state and transaction state.
• Black, isort, flake8 and git diff --check passed for the changed files.

The full Django suite and other database backends have not been run. The benchmark runner, raw samples and full result table are available to attach for reproduction.

Would this slot-clearing approach be acceptable, particularly the retained list allocation and the change visible through aliases to the private queue?

Assistance disclosure:

Codex helped run benchmarks, and draft the description and submission helper.