Code

Opened 9 years ago

Closed 9 years ago

Last modified 7 years ago

#515 closed defect (fixed)

[patch] additional set of cache backends

Reported by: eugene@… Owned by: jacob
Component: Core (Cache system) Version:
Severity: normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

New Backends

Three more backends are introduced:

  • locmem: - simple thread-safe in-memory cache. Good for small web sites.
    • Example: locmem:///.
  • file: - file-based multi-process/thread-safe cache.
    • Example: file:///tmp/django.cache/lazutkin.com/ - creates directory /tmp/django.cache/lazutkin.com and stores cached items as individual files.
  • sql: - sql-based multi-process/thread-safe cache.
    • Example: sql://cache/ --- uses table cache in current database to store cached items.

All three backends accept arguments described in Django's cache framework. Examples: sql://cache/?timeout=60, locmem:///?max_entries=10&cull_frequency=2.

Additionally simple: cache backend's bug is fixed.

Notes:

  • All backends use pickle (cPickle) as means to save the object. I decided against marshal or custom solutions. For now.
  • locmem:
    • Requires reader-writer lock, which is implemented in synch.py (attached). This file should be placed into django/utils directory.
    • Implements simplified (a-la simple:) cache culling strategy, when each cull_frequency item is removed to make room.
  • file:
    • Implements simplified (a-la simple:) cache culling strategy, when each cull_frequency item is removed to make room.
  • sql:
    • When it is time to cull cache, sql: removes expired items first, then removes oldest items by access time to satisfy cull_frequency. This strategy has a drawback: every time cached item is accessed, its access time filed is updated.
    • It can be speeded up with stored procedures.

Missing django-admin Support

The missing part is django-admin support for:

  • creation of cache table for sql: backend
  • creation of cache indices for sql:
  • cleaning up sql: and file: caches

Pseudo model for cache table is:

class Cache(meta.Model):
    # id is assumed
    keyx       = meta.CharField(maxlength=255, unique=True),
    expiration = meta.DateTimeField(db_index=True),
    accessed   = meta.DateTimeField(db_index=True),
    valx       = meta.TextField(), # should be big enough to hold an item

For TextField on MySQL see #489 --- when you create cache table manually make sure to use longtext instead of text.

If you want to use MySQL, use patch from #463. Otherwise you may have strange random errors.

Testing

Notes:

  • I ran all new backends for several days monitoring cache entries.
  • sql: was tested locally using MySQL and SQLite. I use MySQL/MyISAM-based cache on my web site. I didn't try it with PostgreSQL nor MySQL/InnoDB.

Attachments (3)

cache.patch (11.3 KB) - added by eugene@… 9 years ago.
patch for django/core/cache.py
synch.py (2.4 KB) - added by eugene@… 9 years ago.
new file django/utils/synch.py
django.cache.patch (400 bytes) - added by eugene@… 9 years ago.

Download all attachments as: .zip

Change History (15)

Changed 9 years ago by eugene@…

patch for django/core/cache.py

Changed 9 years ago by eugene@…

new file django/utils/synch.py

comment:1 Changed 9 years ago by adrian

Nice! We'll integrate most or all of this...

comment:2 Changed 9 years ago by Simon Willison

This ticket replaces #4.

comment:3 Changed 9 years ago by jacob

(In [686]) Added "locmem" and "file" cache backends. "locmem" is a thread-safe local-memory cache, and "file" is a file-based cache.
This refs #515; much thanks to Eugene Lazutkin!

comment:4 Changed 9 years ago by jacob

  • Resolution set to fixed
  • Status changed from new to closed

(In [692]) Added a database-backed cache backend, along with a tool in django-admin to
create the necessary table structure. This closes #515; thanks again,
Eugene!

comment:5 Changed 9 years ago by eugene@…

  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Type changed from enhancement to defect

Database-backed cache backend doesn't work in MySQL. Reason: 'key' cannot be field name. It is a reserved word. django-admin createcachetable fails. That's why I used goofy 'keyx' in my implementation.

comment:6 Changed 9 years ago by jacob

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [695]) Fixed #515 (again) - renamed "key" field in SQL cache to "cache_key" because
MySQL reserves "key".

comment:7 Changed 9 years ago by eugene@…

  • Resolution fixed deleted
  • Status changed from closed to reopened

Unfortunately changes to my patches removed some exception tolerance from caches. For example:

  • file: I got a coredump when somebody purged /tmp. os.listdir() (line 319) aborted with OSError bringing down Django. It should be a normal situation: directory should be recreated and files should be created like nothing happened. That's how you suppose to clear cache of active web sites.
  • db: I got a coredump when two threads checked cache for an item, and both created it. The second thread aborted citing "insertion failure". It should be absolutely normal situation: just continue to do what you are doing.

Unfortunately current implementations of file: and db: are unusable in real world.

comment:8 Changed 9 years ago by jacob

OK, I'll try to reproduce these bugs and check in a fix. I was unclear why you had the bare except clauses sprinkled around your patch; I'm starting to understand now...

comment:9 Changed 9 years ago by jacob

(In [699]) Added exception handlers to take care of the bugs with the file and db backends
(refs #515). Eugene, I'm going to leave #515 open; can you check the bug fixes
in this revision and mark the ticket as closed if you're satisfied? I don't
run Django in a threaded environment, so I'm having issues reproducing your
errors.

comment:10 Changed 9 years ago by eugene@…

Now the code looks much better. I tried file: cache. Minor one line patch is attached.

Index: cache.py
===================================================================
--- cache.py	(revision 700)
+++ cache.py	(working copy)
@@ -317,6 +317,7 @@
             filelist = os.listdir(self._dir)
         except (IOError, OSError):
             self._createdir()
+            filelist = []
         if len(filelist) > self._max_entries:
             self._cull(filelist)
         try:

I am running db: cache now and so far so good. Last time I got an error immediatelly. Let me run it for a day.

Changed 9 years ago by eugene@…

comment:11 Changed 9 years ago by jacob

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [701]) Quick bug fix to [699] - fixes #515.

comment:12 Changed 9 years ago by jacob

Eugene - feel free to reopen if you find any more bugs. Thanks for your help!

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.