Filebased cache should use the highest pickling protocol

[Bertrand Bordage]

I came across this bug when caching psycopg2._range.NumericRange objects in ​django-cachalot (in order to add Django 1.8 support to it). ​My test suite was failing only on Python 2 and with filebased (and of course, Django 1.8, because django.contrib.postgres didn’t exist before). This error was thrown:

TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

This is because, unlike ​locmem or ​db, ​filebased uses the default pickling protocol and not the highest available. Since we are not trying to achieve read compatibility with old Python versions, we can use the highest protocol here as well.

This issue occur on master, Django 1.7.10, 1.8.4, & 1.9 alpha 1.

[Andrew Artajos]

Hi Bertrand,

I am a new to contributing to Django. Do you have any steps on how to reproduce this?

Nevermind, I managed to reproduce it with this:

>>> from psycopg2.extras import NumericRange
>>> be = NumericRange(lower=300, upper=350)
>>> cache.set('range', be)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/aartajos/Projects/Django/django/django/core/cache/backends/filebased.py", line 58, in set
    f.write(zlib.compress(pickle.dumps(value), -1))
  File "/home/aartajos/.virtualenvs/djangodev2/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

[Bertrand Bordage]

Hi Andrew,

If you need to be able to test it when psycopg2 is not installed, you can also reproduce this bug by creating a class with the same problem:

class UnpickableType(object):
    __slots__ = ('a',)

If you try to pickle an instance of this class, you’ll get the same error.

[Tim Graham]

​PR

[Tim Graham <timograham@…>]

In 48888a1:

Fixed #25501 -- Made the file-based cache backend use the highest pickling protocol.

[Bertrand Bordage]

Great job, thanks :)

[Andrew Artajos]

Thanks for your help Bertrand.

[Jaap Roes]

I've commented on the commit, but I'll do it here as well:

The original intent has always been to use the highest protocol. From help(pickle.dumps):

Specifying a negative protocol version selects the highest protocol version supported.

So this line:

f.write(zlib.compress(pickle.dumps(value), -1))

Was supposed to be:

f.write(zlib.compress(pickle.dumps(value, -1)))

Calling zlib.compress with a compression level of -1 seems to fall back to the default level of 6.

[Tim Graham <timograham@…>]

In 1aba0e4c:

Refs #25501 -- Fixed a typo in django/core/cache/backends/filebased.py

The original intent in refs #20536 was to use the highest protocol.
Calling zlib.compress() with a compression level of -1 seems to
fall back to the default level of 6.