Database and memcached connections break after fork.
|Reported by:||Sebastian Noack||Owned by:||nobody|
|Component:||Database layer (models, ORM)||Version:||1.4|
|Cc:||davidswafford||Triage Stage:||Design decision needed|
|Has patch:||yes||Needs documentation:||no|
|Needs tests:||yes||Patch needs improvement:||no|
If you have a management command that does CPU-heavy tasks, or when implementing a server for certain background tasks, its likely that you will use the multiprocessing module, to scale over multiple CPUs. However django implements connections to the database and memcached as singletons (created on first use, reused forever). So if you have used the database or memcached before forking, the child processes inherit the established connection. And when multiple processes use a connection at the same time (which can and will happen) the requests will fail in an ugly way.
However the multiprocessing module comes with a mechanism provided for such cases, that enables you to cleanup things after fork. My patch uses that mechanism, in order to reset the possibly created database and memcached connections after fork. So that the child process will create its own connection when it needs it.
Change History (8)
Changed 4 years ago by