Opened 4 years ago

Closed 4 years ago

Last modified 9 months ago

#16116 closed New feature (wontfix)

Compress cache data

Reported by: bjourne Owned by: nobody
Component: Core (Cache system) Version: 1.3
Severity: Normal Keywords:
Cc: bjourne@…, lucmult Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: yes UI/UX: no

Description

Often it would be useful if cache data were gzipped. The memcached backend limits key values to 1 mb in size and it is not unreasonable to want to cache a larger html page. Also, html data compresses really well so a 1 mb html file might be just 50 kb compressed which is very important if the amount of memory you have available is limited.

Change History (7)

comment:1 Changed 4 years ago by russellm

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to wontfix
  • Status changed from new to closed

I'm going to mark this wont fix, for two reasons.

Firstly, this is something that should be able to live as an external library. I can't think of any reason that this couldn't be implemented as a caching module.

Secondly, I'm not convinced it is something that is of sufficient general use that it warrants being in the core. Zipping cache data implies a tradeoff of memory for CPU, which will only be of benefit for the case where you are storing large objects in cache.

I'm more than willing to reconsider putting this in core in the future, but that would be as a result of the module existing as a third party option that gains widespread use.

comment:2 Changed 2 years ago by lucmult

  • Cc lucmult added
  • UI/UX unset

Hello guys,

Last week we run on this problem of memcache limitation and started to have much more CPU load on our server due our biggest pages weren't being cached.

python-memcached has support to zip (when zlib is available):

http://bazaar.launchpad.net/~python-memcached-team/python-memcached/trunk/view/head:/memcache.py#L763

I'm willing to submit a patch to MemcachedCache backend to be able to configure it on settings.

settings:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': ['127.0.0.1:11211'],
        'OPTIONS': {
            'CLIENT_OPTIONS': {'server_max_value_length': 67108864, 'pickleProtocol': 2},
            'MIN_COMPRESS_LEN': 500,
        }
    }
}

server_max_value_length is useful because python-memcached uses it to decide if it should ignore the value. It's default to 1024*1024
http://bazaar.launchpad.net/~python-memcached-team/python-memcached/trunk/view/head:/memcache.py#L775

In the docs I would suggest our users to set with the value got from memcache stats:

$ echo "stats settings" | nc localhost 11211`
STAT maxbytes 67108864
...

Although this hasn't shown as the real limit on my local test :(
If the application send a value bigger than memcache can handle it just respond as data is too big to store, no exception is raised, cache is only ignored (as we have currently).

Our usage of MIN_COMPRESS_LEN has shown a significant decrease of network traffic between django server and memcached server.

This fix would benefit more 2 tickets:
Memcached session "forgets" big values - https://code.djangoproject.com/ticket/16358
Memcached using pickle highest protocol - https://code.djangoproject.com/ticket/16378

comment:3 Changed 2 years ago by lucmult

As a side note.

python-memcached has support for mixed values (zipped and not zipped), so it's safe to change this option at any time.

Also, it compress only when bigger than specified on MIN_COMPRESS_LEN, so one concerned by zlib overhead can increase this value, for somebody concerned about network traffic on memcached can make it lower.

comment:4 Changed 2 years ago by lucmult

Pull request with the patch:
https://github.com/django/django/pull/736

Still need docs though.

Is there a way to test it ?

Last edited 2 years ago by lucmult (previous) (diff)

comment:5 Changed 15 months ago by vhermecz

I strongly disagree with the reasons behind closing this request.
While this could live in a 3rd party library, there is actually a memcached cache implementation withing django. And the project should maintain it if it is there.

Regarding lucmult provided a nice patch, would be sweet to pull it...

comment:6 Changed 9 months ago by the_drow

I also disagree.
There is no reason not to support it.
Both python-memcached and pylibmc do support it if zlib is available.
Why shouldn't it be opt-in if required?

comment:7 Changed 9 months ago by timo

If you disagree with a "won't fix" decision, please open a thread on the DevelopersMailingList. Are those requesting this using a 3rd party implementation that has gained widespread use as Russ asked? I think the idea is fine and note #20892 which is another ticket asking for a way to pass options to the memcache backend.

Note: See TracTickets for help on using tickets.
Back to Top