Connecting the same signal multiple times with weak=True allocates new memory that is never deallocated

[phantom-jacob]

We recently observed steadily increasing memory usage in our Django application running under uwsgi with celery workers, which we tracked down to multiple calls to connect the same signal handlers. We eventually figured out that when weak=True on signal connection (the default behavior), each call to connect() allocates new memory (which tracemalloc indicates is largely from weakref) on every call. This memory is not cleared by disconnecting the signal, and in practice we never saw the memory free before the system eventually OOMed.

Anyway, I've developed a standalone reproduction script, which can be run with uv run django_mem_leak.py. If the --weak argument is provided, you can see memory grow drastically:

uv run django_mem_leak.py
Installed 3 packages in 92ms
Running memory audit with 100000 iterations, weak=False
Starting memory usage: 28.88 MiB
[100000/100000] Memory usage: 30.48 MiB
Final memory usage: 30.52 MiB



----------------------------------------------------------------------------------------------------


[ Top 10 memory allocations ]
<frozen importlib._bootstrap_external>:784: size=178 KiB (+178 KiB), count=1678 (+1678), average=109 B
<frozen importlib._bootstrap>:488: size=119 KiB (+119 KiB), count=1379 (+1379), average=89 B
<frozen importlib._bootstrap_external>:801: size=8156 B (+8156 B), count=68 (+68), average=120 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/pathlib/_local.py:59: size=4133 B (+4133 B), count=18 (+18), average=230 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/pathlib/_abc.py:403: size=3595 B (+3595 B), count=10 (+10), average=360 B
<frozen importlib._bootstrap_external>:133: size=3479 B (+3479 B), count=21 (+21), average=166 B
~/.cache/uv/environments-v2/django-mem-leak-5f36d9b360a80206/lib/python3.13/site-packages/django/conf/global_settings.py:433: size=3264 B (+3264 B), count=1 (+1), average=3264 B
<frozen importlib._bootstrap_external>:1704: size=3086 B (+3086 B), count=33 (+33), average=94 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/pathlib/_abc.py:55: size=2915 B (+2915 B), count=11 (+11), average=265 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/pathlib/_local.py:482: size=2802 B (+2802 B), count=13 (+13), average=216 B



uv run django_mem_leak.py --weak
Running memory audit with 100000 iterations, weak=True
Starting memory usage: 27.36 MiB
[100000/100000] Memory usage: 102.73 MiB
Final memory usage: 102.73 MiB



----------------------------------------------------------------------------------------------------


[ Top 10 memory allocations ]
~/.cache/uv/environments-v2/django-mem-leak-5f36d9b360a80206/lib/python3.13/site-packages/django/dispatch/dispatcher.py:120: size=9375 KiB (+9375 KiB), count=200001 (+200001), average=48 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/weakref.py:576: size=7813 KiB (+7813 KiB), count=100001 (+100001), average=80 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/weakref.py:575: size=7812 KiB (+7812 KiB), count=100000 (+100000), average=80 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/weakref.py:582: size=5120 KiB (+5120 KiB), count=1 (+1), average=5120 KiB
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/weakref.py:581: size=2727 KiB (+2727 KiB), count=99743 (+99743), average=28 B
<frozen importlib._bootstrap_external>:784: size=233 KiB (+233 KiB), count=2363 (+2363), average=101 B
<frozen importlib._bootstrap>:488: size=40.9 KiB (+40.9 KiB), count=328 (+328), average=128 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/pathlib/_local.py:59: size=4133 B (+4133 B), count=18 (+18), average=230 B
~/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/pathlib/_abc.py:403: size=3715 B (+3715 B), count=11 (+11), average=338 B
<frozen importlib._bootstrap_external>:133: size=3479 B (+3479 B), count=21 (+21), average=166 B

Use this script to reproduce:

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "django",
# ]
# ///
import argparse
import resource
import sys
import tracemalloc

from django.dispatch import Signal


def signal_handler(sender, **kwargs):
    pass


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Run a memory audit to demonstrate signal handler leaks"
    )
    parser.add_argument(
        "num",
        type=int,
        default=100000,
        nargs="?",
        help="Number of iterations to run the memory audit for",
    )
    parser.add_argument(
        "--weak",
        action="store_true",
        help="Whether to use weak references for audit signals",
    )
    return parser.parse_args()


def _get_memory_usage() -> float:
    maxrss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

    # Mac reports in bytes, Linux reports in KiB
    if sys.platform == "darwin":
        return maxrss / 1024 / 1024

    return maxrss / 1024


def main(
    *,
    num: int,
    weak: bool,
) -> None:
    signal = Signal()

    print(f"Running memory audit with {num} iterations, {weak=}")
    print(f"Starting memory usage: {_get_memory_usage():.2f} MiB")

    tracemalloc.start()
    start_snapshot = tracemalloc.take_snapshot()

    for i in range(1, num + 1):
        print(
            f"[{i}/{num}] Memory usage: {_get_memory_usage():.2f} MiB",
            end="\r",
            flush=True,
        )
        signal.connect(signal_handler, weak=weak, dispatch_uid="test_signal")
        signal.disconnect(signal_handler, dispatch_uid="test_signal")

    print("")
    print(f"Final memory usage: {_get_memory_usage():.2f} MiB")

    end_snapshot = tracemalloc.take_snapshot()
    top_stats = end_snapshot.compare_to(start_snapshot, "lineno")
    print("\n\n")
    print("-" * 100)
    print("\n\n[ Top 10 memory allocations ]")
    for stat in top_stats[:10]:
        print(stat)

    tracemalloc.stop()


if __name__ == "__main__":
    args = parse_args()
    main(**vars(args))