Opened 3 weeks ago
Closed 3 weeks ago
#36957 closed Bug (duplicate)
Django psycopg connection pool + fork()
| Reported by: | Anna | Owned by: | Dhruvan Gnanadhandayuthapani |
|---|---|---|---|
| Component: | Uncategorized | Version: | 6.0 |
| Severity: | Normal | Keywords: | |
| Cc: | Triage Stage: | Unreviewed | |
| Has patch: | yes | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description (last modified by )
Django's PostgreSQL backend stores psycopg_pool.ConnectionPool objects in a class-level dict (DatabaseWrapper._connection_pools). When gunicorn (or any pre-forking server) forks worker processes, all children inherit references to the same pool objects — and crucially, the same underlying TCP sockets to PostgreSQL. Multiple workers then read/write the same socket concurrently, corrupting the PostgreSQL wire protocol.
Root cause
# django/db/backends/postgresql/base.py
class DatabaseWrapper(BaseDatabaseWrapper):
_connection_pools = {} # class-level dict — survives fork()
@property
def pool(self):
if self.alias not in self._connection_pools:
pool = ConnectionPool(...)
self._connection_pools.setdefault(self.alias, pool)
return self._connection_pools[self.alias]
Workaround
# gunicorn.conf.py
def post_fork(server, worker):
from django.db.backends.postgresql.base import DatabaseWrapper
DatabaseWrapper._connection_pools.clear()
Suggested fix
Use os.register_at_fork(after_in_child=...) to clear _connection_pools in child processes, or check os.getpid() in the pool property and recreate when it differs from the creating process.
Tested with
Django 6.0.2
psycopg 3.2.x – 3.3.2
psycopg-pool 3.2.x – 3.3.0
gunicorn 25.x (--worker-class asgi)
Python 3.12 – 3.14
The minimal reproducible example project is in the attachments
Attachments (1)
Change History (8)
by , 3 weeks ago
| Attachment: | django-pool-fork-bug.zip added |
|---|
comment:1 by , 3 weeks ago
| Description: | modified (diff) |
|---|
comment:2 by , 3 weeks ago
| Owner: | set to |
|---|---|
| Status: | new → assigned |
| Triage Stage: | Unreviewed → Accepted |
comment:3 by , 3 weeks ago
| Has patch: | set |
|---|
comment:4 by , 3 weeks ago
comment:5 by , 3 weeks ago
| Triage Stage: | Accepted → Unreviewed |
|---|
Hello Djruvan! The “Accept” status on a Django ticket should be set by a triager after reviewing the ticket. If you haven’t had a chance to read the contribution guidelines, I would appreciate it if you could take a look.
comment:6 by , 3 weeks ago
I would say this is close if not a duplicate of #31637. The problem of forking after connection creation is not specific to psycopg connection pools.
comment:7 by , 3 weeks ago
| Resolution: | → duplicate |
|---|---|
| Status: | assigned → closed |
Agreed, duplicate of #31637.
Opened a PR that implements the
os.register_at_fork()check along with a flag. This solution was preferred over runningos.getpid()in the pool property, because it is wasteful to do it each time the property is accessed.https://github.com/django/django/pull/20790