Code

Opened 7 years ago

Closed 4 years ago

Last modified 3 years ago

#6124 closed (wontfix)

need a non-pickled locmem cache backend

Reported by: lukerl@… Owned by: nobody
Component: Core (Cache system) Version: master
Severity: Keywords: cache, locmem, pickle
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

Rev. 5703 of locmem.py broke some code we have been using. We have a mathematical model consisting of a swig-wrapped C++ object that needs to maintain its state between requests. We initialize the object and store it in the cache, keyed by the user. On each request, we pull the reference to the object and make calculations. Unfortunately, pickling, added in 5703 breaks this code, as our Swig Class is not pickleable. My current workaround has been to remove the pickle.dumps and loads calls wrapping the value to be stored/retrieved. This is not a good long-term solutions. It would be nice to have a sort of "raw" locmem cache that does not attempt to pickle the value as before.

Attachments (1)

locmem.py (4.0 KB) - added by Luke Loeffler <lukerl@…> 7 years ago.

Download all attachments as: .zip

Change History (6)

Changed 7 years ago by Luke Loeffler <lukerl@…>

comment:1 Changed 7 years ago by Luke Loeffler <lukerl@…>

  • Has patch set
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Attached is a patch that allows the user to define the following in settings.py

CACHE_BACKEND = 'locmem:///?nopickle=True'

and get the desired results.

comment:2 Changed 6 years ago by Simon Greenhill <dev@…>

  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Accepted

comment:3 Changed 6 years ago by Pascal Varet <pvaret@…>

Hello Luke,

I can see a few problems with your patch:

  • Well, to start with, it's not a patch. :) You want to give a patch, rather than the patched file itself, for two reasons: one, the file may have received other changes than your own since, so it's important to provide only those lines that you did change, and two, in a patch you can provide changes to several files, which, as you'll see below, is relevent to this case.
  • Your patch should provide documentation for the option you are adding to the locmem cache backend. (That's where you need to provide changes to another file.) On that note, you should absolutely document the fact that your new option breaks the convention that cached object be picklable (see the current cache documentation).
  • Your patch should provide tests. I'm very much not an expert on tests, but I think what you should provide is tests that ensure a non-picklable object is cached when the option nopickle=true is given, and not cached otherwise.
  • Lastly: your patch contains a subtle bug: since you return the same object whenever the cache is called, if any caller modifies the object for its own purpose, then the object is modified for every caller. This is typically bad in the case of cached HTTP responses. See ticket #599, which addressed this very same issue back then. This issue is the reason why pickling was used in the first place, so presumably your patch should provide another, non-pickle-based solution to the same problem.

All this being said, IMHO, your problem (object persistence) is not the one that the Django cache exists to solve. In the Django cache, attempts to cache uncacheable objects fail silently, and cached object eventually expire -- neither are what you need, right?

Perhaps the best solution for you would be to implement your own object persistence. See, what the locmem cache amounts to is a simple old Python dictionary with locks and expiration. Since, unless I'm mistaken, you don't need expiration, perhaps a dictionary somewhere is all you need? The dictionary will be persisted as long as your Django app keeps running, exactly as with the locmem cache backend.

HTH.

comment:4 follow-up: Changed 4 years ago by mtredinnick

  • Resolution set to wontfix
  • Status changed from new to closed

Since using external cache backends has been possible and straightforwards for quite a while now (before Django 1.0, from memory), there's no reason something like this has to be included. If anybody has a reason to use a particular cache implementation, they can do just that and reference it via a path in the settings. There's no need for it to be in Django.

comment:5 in reply to: ↑ 4 Changed 3 years ago by akaihola

Replying to mtredinnick:

Since using external cache backends has been possible and straightforwards for quite a while now (before Django 1.0, from memory), there's no reason something like this has to be included. If anybody has a reason to use a particular cache implementation, they can do just that and reference it via a path in the settings. There's no need for it to be in Django.

See django snippet #2396 for an implementation of a non-pickling locmem external cache backend.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.