Speed up post-commit callback dispatch

[harshit gupta]

Summary:
Speed up post-commit callback dispatch

Description:

BaseDatabaseWrapper.run_and_clear_commit_hooks() drains its detached callback list with pop(0). Each removal shifts the remaining entries, so queue-management overhead is O(N²) in the number of callbacks.

This can matter for bulk work that registers a post-commit callback per item, such as deferred cache updates. Callback work itself may dominate in applications that make network or task-broker calls; those costs are not part of the queue-complexity claim.

A local patch keeps the existing list-detachment step and visits each entry once, clearing its slot before calling it. This makes queue-management overhead O(N), without allocating another queue. Clearing the slot prevents completed callbacks from being retained by the detached list or a saved traceback from a later callback failure.

The existing robust/nonrobust handling stays unchanged. The list allocation still lasts until the frame is released, and failing or unrun callbacks can still be retained by a saved traceback. An external alias to this private list would see None slots instead of a shrinking list, so private-list inspection and direct mutation are not claimed to be identical.

Reproduction and measurements:

Checked against d430084791a61b31cecec132bce3dca19194f7f2 on Python 3.14.5, Windows, SQLite.

The runner compiles the original method from that commit in memory and compares it with the changed method. Its benchmarks include transaction begin/commit, callback registration and callbacks updating an in-memory dictionary. Two workloads give callbacks separately captured bytearrays to check retained state. These are controlled workloads, not production traffic, model writes or broker calls.

There are 48 rows: 12 workloads measured separately on each of four CPU pairs. Each row has five paired timing samples per implementation and five separate tracemalloc measurements. Only two physical cores are allowed at a time, with one process and rotating CPU pairs. All rows, including slower cases, are retained. Different CPU pairs are not pooled. Peaks describe newly traced Python allocations, not total process RAM.

Verification:

• The same 195 focused tests ran on the original and changed methods, with no failures and the same 10 SQLite skips.
• Five regression tests cover exceptions, execution ordering and callback lifetimes, including retained tracebacks and error logs.
• 468 before/after comparisons matched callback order, error/log outcomes, queue state and transaction state.
• Black, isort, flake8 and git diff --check passed for the changed files.

The full Django suite and other database backends have not been run. The benchmark runner, raw samples and full result table are available to attach for reproduction.

Would this slot-clearing approach be acceptable, particularly the retained list allocation and the change visible through aliases to the private queue?

Assistance disclosure:

Codex helped run benchmarks, and draft the description and submission helper.

[David Smith]

It is hard to triage a performance issue without a reproducible benchmark.

Please can you share benchmark scripts along with the claimed performance improvement from this change.

[harshit gupta]

Replying to David Smith:

It is hard to triage a performance issue without a reproducible benchmark.

Please can you share benchmark scripts along with the claimed performance improvement from this change.

| Callbacks.. | Captured payload/callback.. | CPU mask.. | Base time.... | Changed time.. | Speed........ | Base peak... | Changed peak.. | Memory.. |
| ----------- | --------------------------- | ---------- | ------------- | -------------- | -------...... | -----------. | -------------- | -------- |
| 0.......... | 0 B........................ | 0xf....... | 0.0292 ms.... | 0.0461 ms..... | 1.578x slower | 2.406 KiB... | 2.406 KiB..... | same.... |
| 0.......... | 0 B........................ | 0xf0...... | 0.0178 ms.... | 0.0181 ms..... | 1.014x slower | 2.406 KiB... | 2.406 KiB..... | same.... |
| 0.......... | 0 B........................ | 0x300..... | 0.0936 ms.... | 0.0999 ms..... | 1.067x slower | 2.406 KiB... | 2.406 KiB..... | same.... |
| 0.......... | 0 B........................ | 0xc00..... | 0.0970 ms.... | 0.0982 ms..... | 1.012x slower | 2.406 KiB... | 2.406 KiB..... | same.... |
| 1.......... | 0 B........................ | 0xf....... | 0.0329 ms.... | 0.0368 ms..... | 1.118x slower | 2.938 KiB... | 2.938 KiB..... | same.... |
| 1.......... | 0 B........................ | 0xf0...... | 0.0291 ms.... | 0.0307 ms..... | 1.056x slower | 2.938 KiB... | 2.938 KiB..... | same.... |
| 1.......... | 0 B........................ | 0x300..... | 0.1181 ms.... | 0.1092 ms..... | 1.082x faster | 2.938 KiB... | 2.938 KiB..... | same.... |
| 1.......... | 0 B........................ | 0xc00..... | 0.1121 ms.... | 0.1123 ms..... | 1.002x slower | 2.938 KiB... | 2.938 KiB..... | same.... |
| 2.......... | 0 B........................ | 0xf....... | 0.0516 ms.... | 0.0520 ms..... | 1.006x slower | 3.492 KiB... | 3.492 KiB..... | same.... |
| 2.......... | 0 B........................ | 0xf0...... | 0.0544 ms.... | 0.0579 ms..... | 1.064x slower | 3.492 KiB... | 3.492 KiB..... | same.... |
| 2.......... | 0 B........................ | 0x300..... | 0.1270 ms.... | 0.1260 ms..... | 1.008x faster | 3.492 KiB... | 3.492 KiB..... | same.... |
| 2.......... | 0 B........................ | 0xc00..... | 0.1317 ms.... | 0.1247 ms..... | 1.056x faster | 3.492 KiB... | 3.492 KiB..... | same.... |
| 10......... | 0 B........................ | 0xf....... | 0.0767 ms.... | 0.0923 ms..... | 1.204x slower | 8.023 KiB... | 8.023 KiB..... | same.... |
| 10......... | 0 B........................ | 0xf0...... | 0.0749 ms.... | 0.0622 ms..... | 1.205x faster | 8.023 KiB... | 8.023 KiB..... | same.... |
| 10......... | 0 B........................ | 0x300..... | 0.2008 ms.... | 0.2087 ms..... | 1.039x slower | 8.023 KiB... | 8.023 KiB..... | same.... |
| 10......... | 0 B........................ | 0xc00..... | 0.1976 ms.... | 0.1916 ms..... | 1.031x faster | 8.023 KiB... | 8.023 KiB..... | same.... |
| 100........ | 0 B........................ | 0xf....... | 0.3739 ms.... | 0.3492 ms..... | 1.071x faster | 58.664 KiB.. | 58.664 KiB.... | same.... |
| 100........ | 0 B........................ | 0xf0...... | 0.3486 ms.... | 0.3282 ms..... | 1.062x faster | 58.664 KiB.. | 58.664 KiB.... | same.... |
| 100........ | 0 B........................ | 0x300..... | 0.9927 ms.... | 0.9764 ms..... | 1.017x faster | 58.664 KiB.. | 58.664 KiB.... | same.... |
| 100........ | 0 B........................ | 0xc00..... | 0.9607 ms.... | 1.0001 ms..... | 1.041x slower | 58.664 KiB.. | 58.664 KiB.... | same.... |
| 500........ | 0 B........................ | 0xf....... | 1.9030 ms.... | 1.7034 ms..... | 1.117x faster | 291.352 KiB. | 291.352 KiB... | same.... |
| 500........ | 0 B........................ | 0xf0...... | 1.5466 ms.... | 1.5289 ms..... | 1.012x faster | 291.352 KiB. | 291.352 KiB... | same.... |
| 500........ | 0 B........................ | 0x300..... | 4.8458 ms.... | 4.4118 ms..... | 1.098x faster | 291.352 KiB. | 291.352 KiB... | same.... |
| 500........ | 0 B........................ | 0xc00..... | 4.6101 ms.... | 4.3460 ms..... | 1.061x faster | 291.352 KiB. | 291.352 KiB... | same.... |
| 1,000...... | 0 B........................ | 0xf....... | 2.7289 ms.... | 2.9886 ms..... | 1.095x slower | 588.852 KiB. | 588.852 KiB... | same.... |
| 1,000...... | 0 B........................ | 0xf0...... | 4.2581 ms.... | 4.2725 ms..... | 1.003x slower | 588.852 KiB. | 588.852 KiB... | same.... |
| 1,000...... | 0 B........................ | 0x300..... | 10.2119 ms... | 11.7489 ms.... | 1.151x slower | 588.852 KiB. | 588.852 KiB... | same.... |
| 1,000...... | 0 B........................ | 0xc00..... | 9.3432 ms.... | 9.6990 ms..... | 1.038x slower | 588.852 KiB. | 588.852 KiB... | same.... |
| 2,000...... | 0 B........................ | 0xf....... | 13.1922 ms... | 7.3413 ms..... | 1.797x faster | 1181.945 KiB | 1181.945 KiB.. | same.... |
| 2,000...... | 0 B........................ | 0xf0...... | 16.3497 ms... | 9.6548 ms..... | 1.693x faster | 1181.945 KiB | 1181.945 KiB.. | same.... |
| 2,000...... | 0 B........................ | 0x300..... | 21.9616 ms... | 20.7825 ms.... | 1.057x faster | 1181.945 KiB | 1181.945 KiB.. | same.... |
| 2,000...... | 0 B........................ | 0xc00..... | 19.8812 ms... | 20.8758 ms.... | 1.050x slower | 1181.945 KiB | 1181.945 KiB.. | same.... |
| 5,000...... | 0 B........................ | 0xf....... | 60.9961 ms... | 21.3436 ms.... | 2.858x faster | 2964.852 KiB | 2964.852 KiB.. | same.... |
| 5,000...... | 0 B........................ | 0xf0...... | 58.4972 ms... | 21.5353 ms.... | 2.716x faster | 2964.852 KiB | 2964.852 KiB.. | same.... |
| 5,000...... | 0 B........................ | 0x300..... | 58.3460 ms... | 51.6651 ms.... | 1.129x faster | 2964.852 KiB | 2964.852 KiB.. | same.... |
| 5,000...... | 0 B........................ | 0xc00..... | 58.5147 ms... | 50.5312 ms.... | 1.158x faster | 2964.852 KiB | 2964.852 KiB.. | same.... |
| 10,000..... | 0 B........................ | 0xf....... | 205.7161 ms.. | 45.2413 ms.... | 4.547x faster | 5936.820 KiB | 5936.820 KiB.. | same.... |
| 10,000..... | 0 B........................ | 0xf0...... | 192.8131 ms.. | 43.0341 ms.... | 4.480x faster | 5936.820 KiB | 5936.820 KiB.. | same.... |
| 10,000..... | 0 B........................ | 0x300..... | 140.0813 ms.. | 107.9537 ms... | 1.298x faster | 5936.820 KiB | 5936.852 KiB.. | same.... |
| 10,000..... | 0 B........................ | 0xc00..... | 136.9749 ms.. | 120.7018 ms... | 1.135x faster | 5936.820 KiB | 5936.820 KiB.. | same.... |
| 100........ | 65,536 B................... | 0xf....... | 6.7584 ms.... | 5.8872 ms..... | 1.148x faster | 6465.012 KiB | 6465.012 KiB.. | same.... |
| 100........ | 65,536 B................... | 0xf0...... | 4.3995 ms.... | 4.5368 ms..... | 1.031x slower | 6465.012 KiB | 6465.012 KiB.. | same.... |
| 100........ | 65,536 B................... | 0x300..... | 7.0298 ms.... | 6.6261 ms..... | 1.061x faster | 6465.012 KiB | 6465.012 KiB.. | same.... |
| 100........ | 65,536 B................... | 0xc00..... | 6.9050 ms.... | 6.7708 ms..... | 1.020x faster | 6465.012 KiB | 6465.012 KiB.. | same.... |
| 1,000...... | 1,024 B.................... | 0xf....... | 4.9156 ms.... | 4.6207 ms..... | 1.064x faster | 1652.328 KiB | 1652.328 KiB.. | same.... |
| 1,000...... | 1,024 B.................... | 0xf0...... | 5.1891 ms.... | 4.9479 ms..... | 1.049x faster | 1652.328 KiB | 1652.328 KiB.. | same.... |
| 1,000...... | 1,024 B.................... | 0x300..... | 11.8471 ms... | 11.5129 ms.... | 1.029x faster | 1652.328 KiB | 1652.328 KiB.. | same.... |
| 1,000...... | 1,024 B.................... | 0xc00..... | 11.9090 ms... | 11.3502 ms.... | 1.049x faster | 1652.328 KiB | 1652.328 KiB.. | same.... |

[harshit gupta]

The reproducible benchmark script and full performance results requested here have been compiled and submitted in a fresh, fully detailed ticket at #37435.