Support memcached binary protocol in PyLibMCCache

[Mike Tigas]

As per ​http://sendapatch.se/projects/pylibmc/#example-usage , enabling the binary protocol requires the binary=True argument to be passed into pylibmc.Client when it is initialized.

The binary protocol is available in Memcached 1.3+ and provides a performance boost in high-load circumstances: ​http://www.slideshare.net/tmaesaka/memcached-binary-protocol-in-a-nutshell-presentation

pylibmc ignores any unknown options passed into Client.behaviors (see below), so putting this in OPTIONS seems like the way to go.

>>> import pylibmc
>>> a=pylibmc.Client(['127.0.0.1:55838',])
>>> print a.behaviors
{'cas': 0, 'no_block': 0, 'receive_timeout': 0, 'send_timeout': 0, 'ketama_hash': 0, '_poll_timeout': 5000, '_io_bytes_watermark': 66560, 'cache_lookups': 0, '_sort_hosts': 0, '_io_key_prefetch': 0, '_auto_eject_hosts': 0, 'ketama': 0, 'ketama_weighted': 0, '_io_msg_watermark': 500, '_hash_with_prefix_key': 0, 'tcp_nodelay': 0, 'failure_limit': 0, 'buffer_requests': 0, '_socket_send_size': 0, '_retry_timeout': 0, '_noreply': 0, '_socket_recv_size': 0, '_number_of_replicas': 0, 'distribution': 'modula', 'connect_timeout': 4000, 'hash': 'default', 'verify_keys': 0}

>>> a.behaviors = {"binary":True} # should be ignored by pylibmc
>>> print a.behaviors
{'cas': 0, 'no_block': 0, 'receive_timeout': 0, 'send_timeout': 0, 'ketama_hash': 0, '_poll_timeout': 5000, '_io_bytes_watermark': 66560, 'cache_lookups': 0, '_sort_hosts': 0, '_io_key_prefetch': 0, '_auto_eject_hosts': 0, 'ketama': 0, 'ketama_weighted': 0, '_io_msg_watermark': 500, '_hash_with_prefix_key': 0, 'tcp_nodelay': 0, 'failure_limit': 0, 'buffer_requests': 0, '_socket_send_size': 0, '_retry_timeout': 0, '_noreply': 0, '_socket_recv_size': 0, '_number_of_replicas': 0, 'distribution': 'modula', 'connect_timeout': 4000, 'hash': 'default', 'verify_keys': 0}

[Mike Tigas]

Not sure if patch is in the proper format; have also opened a github pull request, tracking this bug. ​https://github.com/django/django/pull/21

Looked into updating the docs, however the documentation appears to leave out implementation-specific details of the different memcached backends. Please let me know if I’m mistaken.

[Jacob]

The patch isn't ideal - special-casing "binary" is a bit annoying - but this is totally worth having.

[Mike Tigas]

I’ll agree with that, though I wasn’t sure how to make this semantically better.

For the older cache framework, django-newcache accepted binary as a cache param (with timeout, cull_frequency, et. al.), with CACHE_BEHAVIORS as a separate settings option. (Using django-newcache as a comparison point since some of the newer 1.3+ cache features like versioning and key prefixing appear to have been based on the newcache implementation.)

But I didn’t quite feel that special casing PyLibMCCache to accept a new base parameter was correct, either …

CACHES = {
    'default' : {
        "BACKEND" : 'django.core.cache.backends.memcached.PyLibMCCache',
        "LOCATION" : '127.0.0.1:11211',
        "BINARY" : True,
        "OPTIONS" : dict(tcp_nodelay=True, ketama=True),
    }
}

… since the description of OPTIONS reads, “Any options that should be passed to cache backend. The list options understood by each backend vary with each backend. […] Cache backends backed by a third-party library will pass their options directly to the underlying cache library.”

In particular, that seems to imply that for consistency’s sake, all implementation-specific options regarding a backend should go into OPTIONS and that it’s up to the backend to do what it needs to provide the correct information to the underlying library.

Technically, a more semantically-correct option would be to do something like:

CACHES = {
    'default' : {
        "BACKEND" : 'django.core.cache.backends.memcached.PyLibMCCache',
        "LOCATION" : '127.0.0.1:11211',
        "OPTIONS" : {
            "binary": True,
            "behaviors" : dict(tcp_nodelay=True, ketama=True),
        }
    }
}

Not really sure what the best patch would be at this point.

[Rob Hudson]

I recently updated django-pylibmc, a 3rd party cache backend, with a few features we wanted at Mozilla. Namely, to make a timeout of zero mean an infinite timeout, add compression support (which pylibmc handles), and support the binary protocol. The package can be found here: ​https://github.com/jbalogh/django-pylibmc.

I'd be happy to help push this forward or bring some code over into Django. Mapping OPTIONS to pylibmc "behaviors" and adding the extra BINARY=True parameter made sense to me. But whichever way is decided, this should be an easy thing to add to this backend.

[Aymeric Augustin]

#19810 is related.

[Bas Peschier]

Enabling binary mode breaks the tests at the moment, because keys with spaces are actually allowed in binary mode.

[Bas Peschier]

Working on it here: ​https://github.com/bpeschier/django/tree/ticket_15815

Still needs some docs to demystify the options for pylibmc a bit. Currently all options are set as behaviors except binary (which still feels a bit ugly)

[otherjacob]

I really don't think special casing is a big deal here -- it's not far off of what BaseCache does already, although I'd rather pop binary off of _options in the _cache method before setting behaviors to it.

Happy to finish this off if bpeschier can't get to it, but it's almost done at this point I think.

[Omer Katz]

So this is already implemented in ​https://github.com/jbalogh/django-pylibmc.
Shouldn't we just merge that cache provider back?

[Omer Katz]

How come this is not supported? What's blocking it exactly?
How can I help?

[Tim Graham]

I don't see a reviewable patch.

[Michał Łowicki]

any news about it?

[Ed Morley]

In case this helps anyone working on this in the future, if you get test failures running the Django pylibmc cache tests with binary mode enabled in test_zero_timeout() - I believe it's due to a bug in older versions of libmemcached-dev.

This appears to only affect .set() and not .add(), and only appears if using binary mode. The bug means that a Django zero timeout (which is converted to -1 when passed to pylibmc) is ignored by libmemcached and the key is in fact set after all.

Affected libmemcached versions include that running on Travis on their Ubuntu precise images (which presuming they are using the official package is ​http://packages.ubuntu.com/precise/libmemcached-dev). libmemcached-dev 1.0.8 on Ubuntu trusty works fine. I couldn't find a related bug or commit on ​https://launchpad.net/libmemcached but Launchpad is pretty painful to use so I may have just missed it.

[Ed Morley]

<obsolete comment>

[Ed Morley]

This has been fixed in ticket #20892, by adding generic support for passing parameters through to the memcached client constructor (and for all backends, not just PyLibMCCache).

pylibmc's constructor is:

    def __init__(self, servers, behaviors=None, binary=False,
                 username=None, password=None):

So on Django master (will be 1.11), binary mode can be enabled using eg:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
        'OPTIONS': {
            'binary': True,
            'username': 'user',
            'password': 'pass',
            'behaviors': {
                'ketama': True,
            }
        }
    }
}

For more examples, see:
​https://docs.djangoproject.com/en/dev/topics/cache/#cache-arguments

I'm going to try and backport these changes (plus ticket #27152) to django-pylibmc, so the same Django settings file will work for both, to make transitioning from "older Django+django-pylibmc backend" to "Django 1.11+ with stock backend" easier.