id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
37033	Psycopg2 has different semantics than psycopg3 and should not be deprecated	Cameron Gorrie		"The context: we have been trialling psycopg3 in our (async-free!) Django 5.2 codebase. I noted that https://docs.djangoproject.com/en/6.0/releases/4.2/#psycopg-3-support says ""Support for psycopg2 is likely to be deprecated and removed at some point in the future."" -- this ticket has my observations that make me believe this is not a good idea.

We have many Celery (5.6.2) tasks that run on idempotent queue workers with Django fully loaded. When we re-deploy, we kill the worker containers and re-queue the tasks that were in-flight / ready to process on that worker. It does this by raising exceptions from signal handlers: https://github.com/celery/billiard/blob/main/billiard/common.py#L106 ultimately raises a `SystemExit` exception from inside the signal handler when it receives the SIGQUIT signal.
 
As outlined in [Safe Asynchronous Exceptions for Python](https://www.cs.williams.edu/~freund/papers/python.pdf), raising exceptions in this way can arbitrarily interrupt execute at any bytecode boundary.

=== Example case

Here's a simplified task handler so you can visualize the problem:

{{{#!python
@celery_app.task(MY_IDEMPOTENT_QUEUE)
def my_task():
    with transaction.atomic():
        [...]
        my_model.save()
        transaction.on_commit(lambda: my_other_task.delay())
}}}

Sometimes, when a signal is delivered while commit() is running, the following happens:

1. PostgreSQL has applied the COMMIT (we can later verify DB state).
2. psycopg3 is interrupted during its Python‑level wait loop; an exception bubbles out of commit() (or _set_autocommit) rather than a normal return.
3. Django sees an exception in `atomic.__exit__` and therefore does not run `on_commit` callbacks, even though the transaction actually committed.

=== Root cause

* psycopg2's COMMIT path uses `PQexec(""COMMIT"")` inside a Py_BEGIN_ALLOW_THREADS region, so the interpreter will not run the Python signal handler until that call returns; this effectively makes COMMIT atomic w.r.t. Python async exceptions.
* psycopg3's design explicitly avoids blocking libpq calls, using a Python generator + wait functions to drive the COMMIT protocol step-by-step. That means Python bytecode is executing throughout, so signal handlers can raise exceptions mid‑protocol.

I do not believe this will be possible to solve with psycopg3's approach, so developers using Django's on_commit inside of celery worker handlers need to be aware of this behaviour."	Uncategorized	new	Uncategorized	6.0	Normal				Unreviewed	0	0	0	0	0	0