Memcached backend closes connection after every request
|Reported by:||Owned by:||Ed Morley|
|Component:||Core (Cache system)||Version:||1.0|
|Cc:||mjmalone@…, daevaorn@…, harm.verhagen+django@…, Rachel Willmer, trbs@…, jeremy.orem@…, emorley@…||Triage Stage:||Accepted|
|Has patch:||yes||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
Fix for http://code.djangoproject.com/ticket/5133 kills servers in production.
The patch takes care of connections kept open, but it introduces another problem - the need to open one or more tcp connections every single request.
With a simple loop, you can make a system run out of sockets easily - after a socket is closed, that port cannot be reused for an eternity, ranging from 1 minute to 4 depending on OS. If enough sockets get stuck in TIME_WAIT state, the server simply fails to connect to memcached and start serving everything from db again - that's not something you want to see on a site with sufficient traffic to need a memcached installation.
In my opinion, the cure is worse than the disease. There's an easy workaround available for the original problem: restart workers after a certain amount of requests. With max-request=500 on a 5 threads deamon process (mod_wsgi, times 20 processes), we never go over 100 connections on our memcached server, started with the default cap of 1024 connections. If you run mod_python, use MaxRequestsPerChild?.
My current solution is to just noop the whole fix with one line in any .py: django.core.cache.backends.memcached.CacheClass?.close = lambda x: None. It might be an idea to make it configurable so people can choose between disconnect after every request and keep it open until process restart.
Change History (15)
comment:12 Changed 20 months ago by
|Patch needs improvement:||set|
comment:13 Changed 5 months ago by
|Owner:||changed from nobody to Ed Morley|
|Status:||new → assigned|
comment:14 Changed 5 months ago by
|Patch needs improvement:||unset|