Psycopg2 has different semantics than psycopg3 and should not be deprecated

[Cameron Gorrie]

The context: we have been trialling psycopg3 in our (async-free!) Django 5.2 codebase. I noted that ​https://docs.djangoproject.com/en/6.0/releases/4.2/#psycopg-3-support says "Support for psycopg2 is likely to be deprecated and removed at some point in the future." -- this ticket has my observations that make me believe this is not a good idea.

We have many Celery (5.6.2) tasks that run on idempotent queue workers with Django fully loaded. When we re-deploy, we kill the worker containers and re-queue the tasks that were in-flight / ready to process on that worker. It does this by raising exceptions from signal handlers: ​https://github.com/celery/billiard/blob/main/billiard/common.py#L106 ultimately raises a SystemExit exception from inside the signal handler when it receives the SIGQUIT signal.

As outlined in [Safe Asynchronous Exceptions for Python](​https://www.cs.williams.edu/~freund/papers/python.pdf), raising exceptions in this way can arbitrarily interrupt execute at any bytecode boundary.

Example case

Here's a simplified task handler so you can visualize the problem:

@celery_app.task(MY_IDEMPOTENT_QUEUE)
def my_task():
    with transaction.atomic():
        [...]
        my_model.save()
        transaction.on_commit(lambda: my_other_task.delay())

Sometimes, when a signal is delivered while commit() is running, the following happens:

1. PostgreSQL has applied the COMMIT (we can later verify DB state).
2. psycopg3 is interrupted during its Python‑level wait loop; an exception bubbles out of commit() (or _set_autocommit) rather than a normal return.
3. Django sees an exception in atomic.__exit__ and therefore does not run on_commit callbacks, even though the transaction actually committed.

Root cause

• psycopg2's COMMIT path uses PQexec("COMMIT") inside a Py_BEGIN_ALLOW_THREADS region, so the interpreter will not run the Python signal handler until that call returns; this effectively makes COMMIT atomic w.r.t. Python async exceptions.
• psycopg3's design explicitly avoids blocking libpq calls, using a Python generator + wait functions to drive the COMMIT protocol step-by-step. That means Python bytecode is executing throughout, so signal handlers can raise exceptions mid‑protocol.

I do not believe this will be possible to solve with psycopg3's approach, so developers using Django's on_commit inside of celery worker handlers need to be aware of this behaviour.