Memcached connections get left open (in certain circumstances).

I have/had the following problem:

With memcached enabled, our site (dms.bildung.hessen.de) does not respond anymore after 10 - 20 minutes. Restarting memcached solves this problem for the next minutes. Therefor it's not a problem with apache or the program logic.

Therefore I tried the patch above. But it not solve our problem.

In the end I use the file-cache with no problems.

I'm sorry, my real email-address is not h.rauch@gmx.net, it's hans.rauch@gmx.net

Please try the latest version of python-memcached (which is 1.4.0 as of this writing). I believe that the error was there and that it has been resolved between 1.3.9 and 1.4.0.

I'm sorry, but I've been using the latest version of memcached (1.2.2) and python-memcached (1.4.0), when I watch these problems.

hans: Indeed, my apologies. I looked at 1.4.0 during the sprint and must have missed it.

here's a patch against python-memcached 1.4.0 that removes threadlocals and doesn't leave any dangling connections for us. The threadlocals bit appeared somewhere in 1.3.x and has been the source for problems for us too. You might also look at cmemcache. This ticket should definitely remain open until we figure something out.

Thanks, your patch seems to solve our problem. Our Site is running for 24 hours and we didn't had to restart memcached or apache.

Improved patch (really just nitpick): http://dpaste.com/hold/41946/

Removing thread locals gives me lots of bad errors after running okay for a few minutes:

Traceback (most recent call last):
 File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 82, in get_response
   response = callback(request, *callback_args, **callback_kwargs)
 File "/usr/lib/python2.5/site-packages/shotserver04/websites/views.py", line 93, in details
   preload_foreign_keys(browsers, browser_group=True)
 File "/usr/lib/python2.5/site-packages/shotserver04/common/object_cache.py", line 165, in preload_foreign_keys
   [getattr(instance, field_id) for instance in instances]))
 File "/usr/lib/python2.5/site-packages/shotserver04/common/object_cache.py", line 92, in get_many
   [KEY_FORMAT % (model._meta.db_table, 'id', id) for id in id_list])
 File "/usr/lib/python2.5/site-packages/django/core/cache/backends/memcached.py", line 41, in get_many
   return self._cache.get_multi(map(smart_str,keys))
 File "/usr/lib/python2.5/site-packages/python_memcached-1.41-py2.5.egg/memcache.py", line 717, in get_multi
   retvals[prefixed_to_orig_key[rkey]] = val   # un-prefix returned key.
KeyError: 'platforms_operatingsystem:id=6'

However, memcached-cleanup-connections.patch seems to fix the open connections problem on my system.

• Django trunk r7519
• memcached 1.1.12-1
• python-memcached-1.41

I just saw this happen on a client's production site after we updated the memcache client from 1.34 to 1.43 (server is 1.1.12). We ended up applying Jacob's original patch because it looked the least intrusive, but it feels like something that python-memcache should be fixing rather than Django having to do manual connection management (my only concern with Jacob's patch). We elected not to go with Matt's patch because it basically undoes something that was included in python-memcache 1.36, which was included to fix some (other) multi-threading issues, according to their ChangeLog.

I've filed a bug upstream with the python memcache providers, so we'll see if anything happens.

Just heard back from the python-memcached maintainer and current behaviour is intended, to allow for more efficient pooling. This problem may well end up being because we're not getting the same thread-id each time inside the same mod_python process or something similar. Since we probably don't want to do inter-thread pooling (for the same reason we don't do database connection pooling), I'd suggest we go with Jacob's original patch for now as a way to manage the connection lifetimes in a well-defined fashion.

Not going to apply this immediately, in case anybody else on the list has bright ideas, but barring any good objections, I think this would be worth applying before 1.0. It's a "bug" in the sense that we're not cooperating well with memcache and problems that only appear when you're wildly successful and/or under high load are not a lot of fun to see.

If python-memcached can't handle being used in a multithreaded environment, which I recollect as the case when I looked at the code many moons ago, it isn't just a mod_python issue, but would also be an issue were mod_wsgi, fastcgi, paste server, cherrypy wsgi server were used in a multithreaded configuration. I suspect that python-memcached may also have a lot of issues if any of those hosting mechanisms were to kill off threads when they feel like they are no longer needed and then create new ones later on. This would possibly be an issue as not sure it would do cleanup of state when a thread no longer becomes used. Anyway, that later point is an issue for python-memcached and not Django unless Django had similar issues with retaining state for threads beyond their lifetime.

(In [8418]) Fixed #5133 -- Explicitly close memcached connections after each request (similar to database connection management). We can't effectively manage the lifecycle by pooling connections and recent versions of python-memcache can lead to connection exhaustion in some quite reasonable setups.