Cache Backend Inconsistencies and Reverse Compatible Failures

[kyle_owens]

When using memcached as the caching server for Django 1.7, a certain test will fail when using one backend but not the other.

# Passes
import memcache
mc = memcache.Client(['127.0.0.1:11211'])
mc.set("test", 3, -1)


# Fails
import pylibmc
mc = pylibmc.Client(["127.0.0.1:11211"])
mc.set("test", 3, -1)

These above snippets are specific examples of what a set would look like in each backend. The python-memcached correctly passes the -1 timeout to the interface of memcached while the pyLibMC passes 18446744073709551615.

This can be witnessed in the memcache log when the verbosity is turned up using -vv in the conf file or options parameters.

This appears to be an error in the way the PyLibMC library handles "-1" as a timeout value.

To demonstrate that this passes with one and fails with the other, please run:

PYTHONPATH=..:$PYTHONPATH python ./runtests.py --settings=test_sqlite cache.tests.MemcachedCacheTests

and between runs, change the settings file to reflect these two options:

# Fails Unit Tests
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
    }

# Passes Unit Tests
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    }

This example was run on an Ubuntu server with memcached installed using:

sudo apt-get install memcached

One change was made to "/etc/memcached.conf" such that "-vv" was uncommented to increase logging output and memcached was restarted.

The line of code where the time out it set is line 57-60 in ./django/core/cache/backends/memcached.py

        elif int(timeout) == 0:
            # Other cache backends treat 0 as set-and-expire. To achieve this
            # in memcache backends, a negative timeout must be passed.
            timeout = -1

This is a traceback from the django unit tests run with the pylibmc backend.

Traceback (most recent call last):
  File "/home/sbsu/django/tests/cache/tests.py", line 508, in test_zero_timeout
    cache.set('key1', 'eggs', 0)
  File "/home/sbsu/django/django/core/cache/backends/memcached.py", line 89, in set
    self._cache.set(key, value, self.get_backend_timeout(timeout))
ClientError: error 9 from memcached_set: CLIENT ERROR

I am currently uncertain of the correct way to fix this issue. The ramifications of this issue are that it is impossible to have immediately expiring cache values through the pylibmc caching backend. The reverse incompatible issues rise from before the patch surrounding issue #22845, the only way of which I'm aware to have non-expiring cache values was to set cache.default_timeout = 0. Doing so in version 1.7 RC 1 will throw exceptions from the memcached server that are unexpected in the django caching framework. This will lead to legacy code being broken.

[Tim Graham]

I got 44 passing tests (1 skipped) with pylibmc 1.3.0. Are you using an older version? Perhaps it's a bug in PyLibMC that's been fixed.

[Tim Graham]

I also tested back to PyLibMC 1.1 and had no problems. Please reopen if can provide more details (or ping me, timograham, in #django-dev). Thanks!

[Tim Graham]

I could reproduce the error on another machine. I believe we should take up the issue with PyLibMC to see if it's a bug on their end. In the meantime, could you please review the doc patch with my understanding of the backwards compatibility concerns.

[kyle_owens]

Reviewed doc patch. Sufficient to prevent issues moving from older versions of django to version 1.7.

[Tim Graham <timograham@…>]

In 8568e7cfa4fb87cd0a9ead1b3fbb7a39d19e98b9:

Added backwards incompatibility note for refs #22845; refs #23082.

Thanks Kyle Owens for the report.

[Tim Graham <timograham@…>]

In 663af270b58f350bcbd9df5cd04ffe69e8ac37c8:

[1.7.x] Added backwards incompatibility note for refs #22845; refs #23082.

Thanks Kyle Owens for the report.

Backport of 8568e7cfa4 from master