Memcached backend closes connection after every request
|Reported by:||booink@…||Owned by:||edmorley|
|Component:||Core (Cache system)||Version:||1.0|
|Cc:||mjmalone@…, daevaorn@…, harm.verhagen+django@…, rwillmer, trbs@…, jeremy.orem@…, emorley@…||Triage Stage:||Accepted|
|Has patch:||yes||Needs documentation:||yes|
|Needs tests:||yes||Patch needs improvement:||yes|
Fix for http://code.djangoproject.com/ticket/5133 kills servers in production.
The patch takes care of connections kept open, but it introduces another problem - the need to open one or more tcp connections every single request.
With a simple loop, you can make a system run out of sockets easily - after a socket is closed, that port cannot be reused for an eternity, ranging from 1 minute to 4 depending on OS. If enough sockets get stuck in TIME_WAIT state, the server simply fails to connect to memcached and start serving everything from db again - that's not something you want to see on a site with sufficient traffic to need a memcached installation.
In my opinion, the cure is worse than the disease. There's an easy workaround available for the original problem: restart workers after a certain amount of requests. With max-request=500 on a 5 threads deamon process (mod_wsgi, times 20 processes), we never go over 100 connections on our memcached server, started with the default cap of 1024 connections. If you run mod_python, use MaxRequestsPerChild?.
My current solution is to just noop the whole fix with one line in any .py: django.core.cache.backends.memcached.CacheClass?.close = lambda x: None. It might be an idea to make it configurable so people can choose between disconnect after every request and keep it open until process restart.
Change History (13)
comment:1 Changed 7 years ago by mmalone
- Cc mjmalone@… added
- Needs documentation unset
- Needs tests unset
- Patch needs improvement unset
comment:3 Changed 7 years ago by Alex
- Triage Stage changed from Unreviewed to Design decision needed
comment:10 Changed 3 years ago by aaugustin
- Triage Stage changed from Design decision needed to Accepted
comment:12 Changed 15 months ago by MarkusH
- Has patch set
- Needs documentation set
- Needs tests set
- Patch needs improvement set