Code

Opened 5 years ago

Closed 3 years ago

Last modified 3 years ago

#11675 closed (fixed)

Support new memcached wrapper pylibmc

Reported by: pippejp@… Owned by: otherjacob
Component: Core (Cache system) Version: master
Severity: Keywords: cache pylibmc memcached.py
Cc: danger@…, tomasz.elendt@…, nikitka@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

pylibmc is a python wrapper around the libmemcache C library.

According to the site, it is faster than cmemcache.

The API is pretty much python-memcached, so I added the fallback: pylibmc -> cmemcache -> memcache.

The patch contains more about eternal caching.

This is my first patch. Thank you.

Attachments (2)

memcached.py.diff (2.2 KB) - added by anonymous 5 years ago.
test.diff (1.4 KB) - added by jammons 4 years ago.
Test for our proposed fix

Download all attachments as: .zip

Change History (23)

Changed 5 years ago by anonymous

comment:1 Changed 5 years ago by pippejp@…

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Sorry, attachment is mine.

comment:2 Changed 5 years ago by Alex

  • Triage Stage changed from Unreviewed to Accepted

I'm marking this accepted, but I don't think implementing *another* fallback is the right way to go. I think the Memcached backend should just have a class "connection_class" attribute and then people can subclass the backend to add support for new memcached wrappers.

comment:3 Changed 5 years ago by stefantalpalaru

This is actually a wrapper around libmemcached: http://tangent.org/552/libmemcached.html
It exposes a lot more functionality than the other ones. Just to name a few: set_multi(), delete_multi(), flush_all(), prepend(), append().

comment:4 Changed 4 years ago by otherjacob

  • Has patch unset
  • Owner changed from nobody to otherjacob
  • Status changed from new to assigned

I'm going to go ahead and adopt Alex's suggestion for future library support.

comment:5 Changed 4 years ago by jammons

So rather than add other fallback we were thinking that one could specify the preferred memcached backend as follows:
CACHE_BACKEND = memcached://server:port?lib=memcache

Where memcache could be cmemcache, memcache, or some other python module. If an engine is not specified, it will default to the current behavior of cmemcache then memcache fallback.

Changed 4 years ago by jammons

Test for our proposed fix

comment:6 Changed 4 years ago by russellm

  • Triage Stage changed from Accepted to Design decision needed

Erm... there's a test, but no actual fix?

Also - I'm not completely sold on either approach. Specifying a module name in the CACHE_BACKEND setting is a neat idea - provided there is some sort of guarantee that the API for a python memcache backend is always the same. From first inspection, this isn't true (set_many vs set_multi). Subclassing seems like overkill, given that there is a limited number of support classes out there.

Another option that hasn't been floated is to allow pylibmc:// definitions that point to a cache.memcache.PyLibMemcache implementation; that is, we do the subclassing, so end users don't have to.

Also - given that cmemcache is clearly deprecated, so we should also deprecate it's use.

comment:7 Changed 4 years ago by apollo13

  • milestone 1.2 deleted

1.2 is in bug fix mode, removing the milestone…

comment:8 Changed 4 years ago by russellm

  • milestone set to 1.2
  • Triage Stage changed from Design decision needed to Accepted

Putting back to 1.2; there has been some discussion about including a partial fix in 1.2 that would start the deprecation process for cmemcache. That won't close this ticket, but it should be addressed before 1.2 final.

To that end, the mailing list discussion also resolved the design decision: we're going to add a new backend for pylibmc.

comment:9 Changed 4 years ago by russellm

  • milestone changed from 1.2 to 1.3

Deferring this ticket to 1.3; #12427 is tracking the deprecation of cmemcache.

comment:10 Changed 4 years ago by mrts

Those who need pylibmc support right now can have a look at http://gist.github.com/334682.

comment:11 Changed 4 years ago by jbergstroem

Will the pylibmc (well, libmemcached) behaviors be exposed? These are very powerful and a game changer in many cases. To mention some of their benefits:

  • consistent hashing
  • switch hash algo to remain compatible with other applications/libraries
  • tweak tcp such as nodelay for a much welcome speed increase

comment:12 follow-up: Changed 4 years ago by mmalone

Hey guys, just a quick comment from my experiences switching to pylibmc on a (non-django) project I'm working on. The transition was pretty smooth, the only thing I noticed that was slightly different in terms of implementation is that pylibmc raises pylibmc.NotFound if you incr/decr a key that doesn't exist while the other Python clients raise ValueError. I'm pretty sure this will break the Django memcache backend for incr/decr operations.

comment:13 in reply to: ↑ 12 Changed 4 years ago by gabrielhurley

Replying to mmalone:

pylibmc raises pylibmc.NotFound if you incr/decr a key that doesn't exist while the other Python clients raise ValueError. I'm pretty sure this will break the Django memcache backend for incr/decr operations.

If necessary, it should be easy enough to catch that and then re-raise a ValueError to be consistent. Django certainly does that to maintain consistency in other places (such as for errors raised by different db backends).

comment:14 Changed 4 years ago by anonymous

  • Cc danger@… added
  • milestone changed from 1.3 to 2.0

comment:15 Changed 4 years ago by anonymous

  • milestone changed from 2.0 to 1.3

comment:16 Changed 3 years ago by elus

  • Cc tomasz.elendt@… added

comment:18 Changed 3 years ago by nmk

  • Cc nikitka@… added

comment:19 Changed 3 years ago by jezdez

  • Has patch set
  • Patch needs improvement set
  • Summary changed from [patch] Support new memcached wrapper pylibmc to Support new memcached wrapper pylibmc

A combined patch with the change and tests is preferred.

comment:20 Changed 3 years ago by russellm

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [15005]) Fixed #11675 -- Added support for the PyLibMC cache library. In order to support this, and clean up some other 1.3 caching additions, this patch also includes some changes to the way caches are defined. This means you can now have multiple caches, in the same way you have multiple databases. A huge thanks to Jacob Burch for the work on the PyLibMC backend, and to Jannis for his work on the cache definition changes.

comment:21 Changed 3 years ago by jacob

  • milestone 1.3 deleted

Milestone 1.3 deleted

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.