[patch] additional set of cache backends

[eugene@…]

New Backends

Three more backends are introduced:

• locmem: - simple thread-safe in-memory cache. Good for small web sites.
  • Example: locmem:///.
• file: - file-based multi-process/thread-safe cache.
  • Example: file:///tmp/django.cache/lazutkin.com/ - creates directory /tmp/django.cache/lazutkin.com and stores cached items as individual files.
• sql: - sql-based multi-process/thread-safe cache.
  • Example: sql://cache/ --- uses table cache in current database to store cached items.

All three backends accept arguments described in ​Django's cache framework. Examples: sql://cache/?timeout=60, locmem:///?max_entries=10&cull_frequency=2.

Additionally simple: cache backend's bug is fixed.

Notes:

• All backends use pickle (cPickle) as means to save the object. I decided against marshal or custom solutions. For now.
• locmem:
  • Requires reader-writer lock, which is implemented in synch.py (attached). This file should be placed into django/utils directory.
  • Implements simplified (a-la simple:) cache culling strategy, when each cull_frequency item is removed to make room.
• file:
  • Implements simplified (a-la simple:) cache culling strategy, when each cull_frequency item is removed to make room.
• sql:
  • When it is time to cull cache, sql: removes expired items first, then removes oldest items by access time to satisfy cull_frequency. This strategy has a drawback: every time cached item is accessed, its access time filed is updated.
  • It can be speeded up with stored procedures.

Missing django-admin Support

The missing part is django-admin support for:

• creation of cache table for sql: backend
• creation of cache indices for sql:
• cleaning up sql: and file: caches

Pseudo model for cache table is:

class Cache(meta.Model):
    # id is assumed
    keyx       = meta.CharField(maxlength=255, unique=True),
    expiration = meta.DateTimeField(db_index=True),
    accessed   = meta.DateTimeField(db_index=True),
    valx       = meta.TextField(), # should be big enough to hold an item

For TextField on MySQL see #489 --- when you create cache table manually make sure to use longtext instead of text.

If you want to use MySQL, use patch from #463. Otherwise you may have strange random errors.

Testing

Notes:

• I ran all new backends for several days monitoring cache entries.
• sql: was tested locally using MySQL and SQLite. I use MySQL/MyISAM-based cache on my web site. I didn't try it with PostgreSQL nor MySQL/InnoDB.

[eugene@…]

patch for django/core/cache.py

[eugene@…]

new file django/utils/synch.py

[Adrian Holovaty]

Nice! We'll integrate most or all of this...

[Simon Willison]

This ticket replaces #4.

[Jacob]

(In [686]) Added "locmem" and "file" cache backends. "locmem" is a thread-safe local-memory cache, and "file" is a file-based cache.
This refs #515; much thanks to Eugene Lazutkin!

[Jacob]

(In [692]) Added a database-backed cache backend, along with a tool in django-admin to
create the necessary table structure. This closes #515; thanks again,
Eugene!

[eugene@…]

Database-backed cache backend doesn't work in MySQL. Reason: 'key' cannot be field name. It is a reserved word. django-admin createcachetable fails. That's why I used goofy 'keyx' in my implementation.

[Jacob]

(In [695]) Fixed #515 (again) - renamed "key" field in SQL cache to "cache_key" because
MySQL reserves "key".

[eugene@…]

Unfortunately changes to my patches removed some exception tolerance from caches. For example:

• file: I got a coredump when somebody purged /tmp. os.listdir() (line 319) aborted with OSError bringing down Django. It should be a normal situation: directory should be recreated and files should be created like nothing happened. That's how you suppose to clear cache of active web sites.

• db: I got a coredump when two threads checked cache for an item, and both created it. The second thread aborted citing "insertion failure". It should be absolutely normal situation: just continue to do what you are doing.

Unfortunately current implementations of file: and db: are unusable in real world.

[Jacob]

OK, I'll try to reproduce these bugs and check in a fix. I was unclear why you had the bare except clauses sprinkled around your patch; I'm starting to understand now...

[Jacob]

(In [699]) Added exception handlers to take care of the bugs with the file and db backends
(refs #515). Eugene, I'm going to leave #515 open; can you check the bug fixes
in this revision and mark the ticket as closed if you're satisfied? I don't
run Django in a threaded environment, so I'm having issues reproducing your
errors.

[eugene@…]

Now the code looks much better. I tried file: cache. Minor one line patch is attached.

Index: cache.py
===================================================================
--- cache.py	(revision 700)
+++ cache.py	(working copy)
@@ -317,6 +317,7 @@
             filelist = os.listdir(self._dir)
         except (IOError, OSError):
             self._createdir()
+            filelist = []
         if len(filelist) > self._max_entries:
             self._cull(filelist)
         try:

I am running db: cache now and so far so good. Last time I got an error immediatelly. Let me run it for a day.

[Jacob]

(In [701]) Quick bug fix to [699] - fixes #515.

[Jacob]

Eugene - feel free to reopen if you find any more bugs. Thanks for your help!