Opened 3 weeks ago

Closed 3 weeks ago

#36957 closed Bug (duplicate)

Django psycopg connection pool + fork()

Reported by: Anna Owned by: Dhruvan Gnanadhandayuthapani
Component: Uncategorized Version: 6.0
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Anna)

Django's PostgreSQL backend stores psycopg_pool.ConnectionPool objects in a class-level dict (DatabaseWrapper._connection_pools). When gunicorn (or any pre-forking server) forks worker processes, all children inherit references to the same pool objects — and crucially, the same underlying TCP sockets to PostgreSQL. Multiple workers then read/write the same socket concurrently, corrupting the PostgreSQL wire protocol.

Root cause

# django/db/backends/postgresql/base.py
class DatabaseWrapper(BaseDatabaseWrapper):
    _connection_pools = {}   # class-level dict — survives fork()

    @property
    def pool(self):
        if self.alias not in self._connection_pools:
            pool = ConnectionPool(...)
            self._connection_pools.setdefault(self.alias, pool)
        return self._connection_pools[self.alias]

Workaround

# gunicorn.conf.py
def post_fork(server, worker):
    from django.db.backends.postgresql.base import DatabaseWrapper
    DatabaseWrapper._connection_pools.clear()

Suggested fix
Use os.register_at_fork(after_in_child=...) to clear _connection_pools in child processes, or check os.getpid() in the pool property and recreate when it differs from the creating process.

Tested with
Django 6.0.2
psycopg 3.2.x – 3.3.2
psycopg-pool 3.2.x – 3.3.0
gunicorn 25.x (--worker-class asgi)
Python 3.12 – 3.14

The minimal reproducible example project is in the attachments

Attachments (1)

django-pool-fork-bug.zip (11.6 KB ) - added by Anna 3 weeks ago.

Download all attachments as: .zip

Change History (8)

by Anna, 3 weeks ago

Attachment: django-pool-fork-bug.zip added

comment:1 by Anna, 3 weeks ago

Description: modified (diff)

comment:2 by Dhruvan Gnanadhandayuthapani, 3 weeks ago

Owner: set to Dhruvan Gnanadhandayuthapani
Status: newassigned
Triage Stage: UnreviewedAccepted

comment:3 by Dhruvan Gnanadhandayuthapani, 3 weeks ago

Has patch: set

comment:4 by Dhruvan Gnanadhandayuthapani, 3 weeks ago

Opened a PR that implements the os.register_at_fork() check along with a flag. This solution was preferred over running os.getpid() in the pool property, because it is wasteful to do it each time the property is accessed.

https://github.com/django/django/pull/20790

comment:5 by JaeHyuckSa, 3 weeks ago

Triage Stage: AcceptedUnreviewed

Hello Djruvan! The “Accept” status on a Django ticket should be set by a triager after reviewing the ticket. If you haven’t had a chance to read the contribution guidelines, I would appreciate it if you could take a look.

Last edited 3 weeks ago by JaeHyuckSa (previous) (diff)

comment:6 by Simon Charette, 3 weeks ago

I would say this is close if not a duplicate of #31637. The problem of forking after connection creation is not specific to psycopg connection pools.

comment:7 by Mariusz Felisiak, 3 weeks ago

Resolution: duplicate
Status: assignedclosed

Agreed, duplicate of #31637.

Note: See TracTickets for help on using tickets.
Back to Top