Opened 4 years ago

Last modified 18 months ago

#12982 new New feature

Add get_or_set method to cache API

Reported by: Alex Owned by: snow0x2d0
Component: Core (Cache system) Version: master
Severity: Normal Keywords: cache
Cc: selwin@…, snow0x2d0@…, mumino Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no


Add a cache.get_or_set() method to Django's cache abstraction, this takes the same arguments as get() except the 2nd argument is a callable, simple implementation looks like:

def get_or_set(self, key, func, **kwargs):
    val = self.get(key)
    if val is None:
        val = func()
        self.set(key, val)
    return val

Attachments (2)

get_or_set.diff (2.7 KB) - added by snow0x2d0 3 years ago.
get_or_set2.diff (3.5 KB) - added by mumino 19 months ago.

Download all attachments as: .zip

Change History (18)

comment:1 Changed 4 years ago by petrilli

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

There's a big race condition in here, unfortunately. If you're using caching (and you should be), and multiple threads access the cache at the same time, then you end up with them over-writing one another. Depending on how this is used, it could be a big issue, or not. If you're talking to memcached, then what you want is add/replace or append/prepend. If you really want to use this approach, then you'll need to use cas, which is "check and set".

I don't believe that these are exposed at this point, unfortunately.

comment:2 Changed 4 years ago by russellm

  • Triage Stage changed from Unreviewed to Accepted

The race condition is a problem, but as Alex noted on IRC, we should be able to avoid this by doing an add() rather than a set().

comment:3 Changed 4 years ago by anonymous

  • Cc selwin@… added

comment:4 Changed 3 years ago by jmeed

  • Owner changed from nobody to jmeed
  • Status changed from new to assigned

comment:5 Changed 3 years ago by snow0x2d0

  • Cc snow0x2d0@… added

comment:6 Changed 3 years ago by snow0x2d0

Is work still ongoing on this? If not, I have a patch, docs and tests waiting in the wings.

Changed 3 years ago by snow0x2d0

comment:7 Changed 3 years ago by snow0x2d0

  • Has patch set

Not sure how else to go about contacting jmeed. I've gone ahead and added my patch, hopefully I'm not stepping on any toes.

comment:8 Changed 3 years ago by jezdez

  • milestone changed from 1.3 to 1.4
  • Summary changed from Pony: cache.get_or_set() to Add get_or_set method to cache API

Removing 1.3 milestone since we are way over the feature freeze.

comment:9 Changed 3 years ago by julien

  • Component changed from Core framework to Cache system

comment:10 Changed 3 years ago by lukeplant

  • Type set to New feature

comment:11 Changed 3 years ago by lukeplant

  • Severity set to Normal

comment:12 Changed 3 years ago by ptone

  • Easy pickings unset
  • Owner changed from jmeed to snow0x2d0
  • Status changed from assigned to new
  • Triage Stage changed from Accepted to Ready for checkin
  • UI/UX unset

patch applies to rev 16843
Docs are clear
tests look pretty thorough

comment:13 Changed 3 years ago by Alex

  • Patch needs improvement set
  • Triage Stage changed from Ready for checkin to Accepted

Moving back to accepted, there are a few issues: 1) in the docs, get_or_set() should show its return values, 2) get_or_set() should *only* take a callable second argument, it's too easy to get the parameters wrong otherwise, 3) it should handle when add() raises an error (which it will if the key is already set).

Thanks for working on this ticket!

comment:14 Changed 3 years ago by jacob

  • milestone 1.4 deleted

Milestone 1.4 deleted

Changed 19 months ago by mumino

comment:15 Changed 19 months ago by mumino

  • Cc mumino added

I made a patch based on snow0x2d0's patch.
my proposal is a little different signature for get_or_set:

   cached_data, is_set = cache.get_or_set(key, default, timeout=None, version=None)

maybe we shouldn't force second parameter be callable. I prefer

cache.get_or_set(score, 0}


cache.get_or_set(score, lambda: 0}

comment:16 Changed 18 months ago by snow0x2d0

Wow, I completely missed the updates to this apparently, my apologizes.

I created two topic github branches:

They both address two of Alex's concerns. Documentation has been updated to show the return values and an additional condition has been added to catch when add fails if a key exists. From looking over the other backend code it looks like in most cases add() will return False if a key exists already (this is the case for both memcache backends). In the case of database backend an add() with an existing key simply functions like a set().

Personally I don't know that there is likely to be confusion over the order of arguments. The order of parameters is just like the get() function and also follows the function name (namely get, then set). That being the case the primary difference between the two branches above is that the ticket_12982_b branch the second parameter _may_ be callable and in the ticket_12982 branch the second argument _must_ be callable (and a typeerror is thrown if not).

Both branches have been rebased to the current master commit (as of about 6am CST 10-29-12) and all tests pass. I can submit a proper pull request for either once approved.

Add Comment

Modify Ticket

Change Properties
<Author field>
as new
The owner will be changed from snow0x2d0 to anonymous. Next status will be 'assigned'
as The resolution will be set. Next status will be 'closed'

E-mail address and user name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.