Opened 2 months ago

Last modified 2 months ago

#31637 new New feature

Registering database connections for cleanup on fork

Reported by: Aarni Koskela Owned by: nobody
Component: Database layer (models, ORM) Version: 3.0
Severity: Normal Keywords:
Cc: Simon Charette Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When using multiprocessing.Pool() or other process-forking APIs, one might bump into

django.db.utils.OperationalError: SSL error: decryption failed or bad record mac

or similar inconsistency errors in the child processes, since the socket is passed down into the forked process.

The quick fix is to

from django import db
db.connections.close_all()

before forking.

Python 3.7 introduced the os.register_at_fork function: https://docs.python.org/3.8/library/os.html#os.register_at_fork

It could be a good idea for Django to use this function to register database connections to be discarded (not cleanly closed, just dropped, as far as a forked process is concerned!) in forked child processes? That way the parent process could use established connection state as before.

Change History (1)

comment:1 Changed 2 months ago by Simon Charette

Cc: Simon Charette added
Triage Stage: UnreviewedAccepted

I think we should do this even if we're moving away from fork'ing internally (e.g. parallel test runner moving to spawn) as it's a really common pitfall when using multiprocessing with the ORM.

Note: See TracTickets for help on using tickets.
Back to Top