Opened 6 years ago

Closed 3 years ago

#11011 closed New feature (duplicate)

Memcached backend: pass parameters to memcache backend

Reported by: erny Owned by: otherjacob
Component: Core (Cache system) Version: 1.0
Severity: Normal Keywords: memcached
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


The problem:

  1. Django instantiates memcache client w/o parameters, i.e. pickleProtocol=0
  2. I compress a string to < 512K, binary pickle format
  3. I encapsulate it inside a dummy object to prevent getting it converted to unicode
  4. I pass it to cache.set
  5. django does the set w/o min_compress_len.
  6. Then python-memcache.set does (indirectly) o dumps with pickleProtocol=0, which results in a my < 512K to a >1,4M string and is silently discarted as max is 1 MB.

Possible solutions:

  • django memcached passes pickleProtocol option to memcached backend client
  • django memcached does implement splitting into chunks
  • I split it into smaller parts in the dummy objects

Change History (8)

comment:1 Changed 6 years ago by Alex

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 5 years ago by mtredinnick

A CACHE_OPTIONS setting, similar to DATABASE_OPTIONS, might be appropriate here. I can imagine that different settings backends requiring a number of different settings (long-term) and trying to jam them all in a URI gets ugly fast. I wouldn't stand in the way of a new setting here (the options would be CACHE_BACKEND specific, but that's fine when you're fine-tuning backend params).

comment:3 Changed 5 years ago by otherjacob

  • milestone set to 1.4
  • Owner changed from nobody to otherjacob

This should be doable, but probably too late to get into 1.3. I'll see if I can come up with a good fix for 1.4

comment:4 Changed 4 years ago by julien

  • Severity set to Normal
  • Type set to New feature

comment:5 Changed 4 years ago by jacob

  • milestone 1.4 deleted

Milestone 1.4 deleted

comment:2 Changed 4 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:3 Changed 4 years ago by aaugustin

  • Easy pickings unset

Change Easy pickings from NULL to False.

comment:4 Changed 3 years ago by aaugustin

  • Resolution set to duplicate
  • Status changed from new to closed
  • #19810 proposes using the highest pickle protocol available, which is going to resolve the root cause — pickling with an inefficient protocol.
  • The CACHES setting was already split like DATABASES, as suggested by Malcolm in comment 2.

I think there's nothing left in this ticket, closing as a duplicate of #19810 which is more specific and will resolve the problem.

Note: See TracTickets for help on using tickets.
Back to Top