Compress cache data

[bjourne]

Often it would be useful if cache data were gzipped. The memcached backend limits key values to 1 mb in size and it is not unreasonable to want to cache a larger html page. Also, html data compresses really well so a 1 mb html file might be just 50 kb compressed which is very important if the amount of memory you have available is limited.

[Russell Keith-Magee]

I'm going to mark this wont fix, for two reasons.

Firstly, this is something that should be able to live as an external library. I can't think of any reason that this couldn't be implemented as a caching module.

Secondly, I'm not convinced it is something that is of sufficient general use that it warrants being in the core. Zipping cache data implies a tradeoff of memory for CPU, which will only be of benefit for the case where you are storing large objects in cache.

I'm more than willing to reconsider putting this in core in the future, but that would be as a result of the module existing as a third party option that gains widespread use.

[lucmult]

Hello guys,

Last week we run on this problem of memcache limitation and started to have much more CPU load on our server due our biggest pages weren't being cached.

python-memcached has support to zip (when zlib is available):

​http://bazaar.launchpad.net/~python-memcached-team/python-memcached/trunk/view/head:/memcache.py#L763

I'm willing to submit a patch to MemcachedCache backend to be able to configure it on settings.

settings:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': ['127.0.0.1:11211'],
        'OPTIONS': {
            'CLIENT_OPTIONS': {'server_max_value_length': 67108864, 'pickleProtocol': 2},
            'MIN_COMPRESS_LEN': 500,
        }
    }
}

server_max_value_length is useful because python-memcached uses it to decide if it should ignore the value. It's default to 1024*1024
​http://bazaar.launchpad.net/~python-memcached-team/python-memcached/trunk/view/head:/memcache.py#L775

In the docs I would suggest our users to set with the value got from memcache stats:

$ echo "stats settings" | nc localhost 11211`
STAT maxbytes 67108864
...

Although this hasn't shown as the real limit on my local test :(
If the application send a value bigger than memcache can handle it just respond as data is too big to store, no exception is raised, cache is only ignored (as we have currently).

Our usage of MIN_COMPRESS_LEN has shown a significant decrease of network traffic between django server and memcached server.

This fix would benefit more 2 tickets:
Memcached session "forgets" big values - https://code.djangoproject.com/ticket/16358
Memcached using pickle highest protocol - https://code.djangoproject.com/ticket/16378

[lucmult]

As a side note.

python-memcached has support for mixed values (zipped and not zipped), so it's safe to change this option at any time.

Also, it compress only when bigger than specified on MIN_COMPRESS_LEN, so one concerned by zlib overhead can increase this value, for somebody concerned about network traffic on memcached can make it lower.

[lucmult]

Pull request with the patch:
​https://github.com/django/django/pull/736

Still need docs though.

Is there a way to test it ?

[vhermecz]

I strongly disagree with the reasons behind closing this request.
While this could live in a 3rd party library, there is actually a memcached cache implementation withing django. And the project should maintain it if it is there.

Regarding lucmult provided a nice patch, would be sweet to pull it...

[Omer Katz]

I also disagree.
There is no reason not to support it.
Both python-memcached and pylibmc do support it if zlib is available.
Why shouldn't it be opt-in if required?

[Tim Graham]

If you disagree with a "won't fix" decision, please open a thread on the DevelopersMailingList. Are those requesting this using a 3rd party implementation that has gained widespread use as Russ asked? I think the idea is fine and note #20892 which is another ticket asking for a way to pass options to the memcache backend.