need a non-pickled locmem cache backend

[lukerl@…]

Rev. 5703 of locmem.py broke some code we have been using. We have a mathematical model consisting of a swig-wrapped C++ object that needs to maintain its state between requests. We initialize the object and store it in the cache, keyed by the user. On each request, we pull the reference to the object and make calculations. Unfortunately, pickling, added in 5703 breaks this code, as our Swig Class is not pickleable. My current workaround has been to remove the pickle.dumps and loads calls wrapping the value to be stored/retrieved. This is not a good long-term solutions. It would be nice to have a sort of "raw" locmem cache that does not attempt to pickle the value as before.

[Luke Loeffler <lukerl@…>]

Attached is a patch that allows the user to define the following in settings.py

CACHE_BACKEND = 'locmem:///?nopickle=True'

and get the desired results.

[Pascal Varet <pvaret@…>]

Hello Luke,

I can see a few problems with your patch:

• Well, to start with, it's not a patch. :) You want to give a patch, rather than the patched file itself, for two reasons: one, the file may have received other changes than your own since, so it's important to provide only those lines that you did change, and two, in a patch you can provide changes to several files, which, as you'll see below, is relevent to this case.

• Your patch should provide documentation for the option you are adding to the locmem cache backend. (That's where you need to provide changes to another file.) On that note, you should absolutely document the fact that your new option breaks the convention that cached object be picklable (see the current cache documentation).

• Your patch should provide tests. I'm very much not an expert on tests, but I think what you should provide is tests that ensure a non-picklable object is cached when the option nopickle=true is given, and not cached otherwise.

• Lastly: your patch contains a subtle bug: since you return the same object whenever the cache is called, if any caller modifies the object for its own purpose, then the object is modified for every caller. This is typically bad in the case of cached HTTP responses. See ticket #599, which addressed this very same issue back then. This issue is the reason why pickling was used in the first place, so presumably your patch should provide another, non-pickle-based solution to the same problem.

All this being said, IMHO, your problem (object persistence) is not the one that the Django cache exists to solve. In the Django cache, attempts to cache uncacheable objects fail silently, and cached object eventually expire -- neither are what you need, right?

Perhaps the best solution for you would be to implement your own object persistence. See, what the locmem cache amounts to is a simple old Python dictionary with locks and expiration. Since, unless I'm mistaken, you don't need expiration, perhaps a dictionary somewhere is all you need? The dictionary will be persisted as long as your Django app keeps running, exactly as with the locmem cache backend.

HTH.

[Malcolm Tredinnick]

Since using external cache backends has been possible and straightforwards for quite a while now (before Django 1.0, from memory), there's no reason something like this has to be included. If anybody has a reason to use a particular cache implementation, they can do just that and reference it via a path in the settings. There's no need for it to be in Django.

[Antti Kaihola]

Replying to mtredinnick:

Since using external cache backends has been possible and straightforwards for quite a while now (before Django 1.0, from memory), there's no reason something like this has to be included. If anybody has a reason to use a particular cache implementation, they can do just that and reference it via a path in the settings. There's no need for it to be in Django.

See ​django snippet #2396 for an implementation of a non-pickling locmem external cache backend.