cmemcache and python-memcache hashing differences

[Ludvig Ericson]

There needs to be support for decidedly using either only python-memcached or cmemcache.

The reason being that the two libraries hash different ways - if you have a box with cmemcache and two without, one of those boxes will store sessions on a different memcached than the other two.

I'm not sure what to do about it, but Malcolm wants documentation and/or a setting with which you can tell Django "use only this library" and fail if it isn't found.

ML thread: ​http://groups.google.com/group/django-developers/t/307825c92ca5bffa

Arguably, this should be something that the two libraries come together on, but at least solving it on our part would be nice.

It might be relevant that I've made a hackesque module which monkey patches python-memcached to hash the same way as cmemcache, see ​http://pypi.python.org/pypi/cmemcache_hash

[taleinat]

What about having Django's memcached cache backend implement its own
hashing algorithm? This could be the "standard" one used in
libmemcache and cmemcache, or perhaps a consistent hashing algorithm
[1] such as libketama[2].

[1]​http://www.socialtext.net/memcached/index.cgi?
faq#what_is_a_consistent_hashing_client

[2]​http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-
_a_consistent_hashing_algo_for_memcache_clients

[taleinat]

What about having Django's memcached cache backend implement its own hashing algorithm? This could be the "standard" one used in libmemcache and cmemcache, or perhaps a ​consistent hashing algorithm such as ​libketama.

[Ludvig Ericson]

Another, new algorithm is exactly what we don't need. But I don't think that was what you were suggesting.

As for ketama hashing, ketama is in no way the only consistent hashing algorithm out there and while Python bindings exist, I'm more in favor of waiting for the memcached libraries themselves to get this functionality. Though some have, like ​pylibmc.

I think the best thing would be for Django to:

1. Offer the alternative of forcing usage of one library.
2. Default to trying to import python-memcached with ​cmemcache_hash so as to interoperate with other libraries.

And lastly, I think the mailing list is the preferred medium for discussion. :-)

[Ludvig Ericson]

Good news, everyone! The maintainer of python-memcached said he'd include my cmemcache_hash with the distribution or something, so this will no longer be as dangerous as it was.

Question is, is it still desirable to be able to explicitly set one or the other? I mean cmemcache can kill FastCGI workers and stuff... Seen it happen. I would still appreciate the option.

[anonymous]

I think we can resolve this as invalid for 1.4 and wontfix for 1.3 with the deprecation of cmemcache and the requirement for explicit backend setting.

[otherjacob]

Replying to anonymous:

I think we can resolve this as invalid for 1.4 and wontfix for 1.3 with the deprecation of cmemcache and the requirement for explicit backend setting.

Also, that would be me not realizing I wasn't logged in.

[otherjacob]

cmemcache is being deprecated in 1.3 and removed in 1.4.