Opened 17 years ago

Last modified 5 months ago

#5815 assigned New feature

Adds per-view cache refreshing (clearing)

Reported by: k0001 Owned by: Ahter Sönmez
Component: Core (Cache system) Version: dev
Severity: Normal Keywords: cache refresh clear
Cc: Petr Přikryl Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

This patch adds the possibility to clear(refresh) cached data per view.

SmileyChris sugested it would be good to add another key to the cached keys where we could keep track of all the different cache_page_key hashes, so we that can delete those then.

It is possible to clear the cache for an specific view, to achieve that, you
should tell your view's url path (and optionally, a key_prefix) to
clear_cache_for_path(path, key_prefix=None).

    from django.utils.cache import clear_cache_for_path
    
    # clear the all cached data for path '/blog/posts/2/'
    clear_cache_for_path('/blog/posts/2/')

Note that it will delete every page cached from matching this path, Vary headers
doesn't matter.

NOTE: Thanks SmileyChris for your help and supervision with this.

Attachments (1)

add_cache_clear_feature.diff (3.7 KB ) - added by k0001 17 years ago.
adds per-view clear cache support

Download all attachments as: .zip

Change History (18)

by k0001, 17 years ago

adds per-view clear cache support

comment:1 by Simon G., 17 years ago

Triage Stage: UnreviewedDesign decision needed

comment:2 by Jacob, 17 years ago

Needs tests: set
Patch needs improvement: set
Triage Stage: Design decision neededAccepted

I like the idea. However, I'm -1 on the implementation -- adding another key to the cache seems unnecessary.

comment:3 by Chris Beaven, 17 years ago

I don't see how we can get around adding a key to index the view's cache keys. Since they are hashed, there's no way of retrieving them with the current cache API (you'd need some type of .get_starts_with() method which I'm not sure would be even possible)

comment:4 by Julian Bez, 16 years ago

One could argue that you are doing something wrong if you need per-site or per-view cache clearing.

If you have a page that changes more than once in a month, you are better off using the cache template tag. This allows you to share cached content between views and the keys can be deleted more easily when something got updated. With per-view caching you always cache the whole page, for every page. Imagine how often you cache your header or footer that way if you have a larger site.

comment:5 by Gabriel Hurley, 14 years ago

Severity: Normal
Type: New feature

comment:6 by Aymeric Augustin, 13 years ago

UI/UX: unset

Change UI/UX from NULL to False.

comment:7 by Aymeric Augustin, 13 years ago

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:8 by Aymeric Augustin, 12 years ago

Triage Stage: AcceptedDesign decision needed

Setting to DDN since there's some disagreement between Jacob and Chris.

Explicit cache invalidation can be useful in some scenarios, it'd be nice to support this.

comment:9 by Jacob, 12 years ago

Triage Stage: Design decision neededAccepted

If there's really not a better implementation (perhaps there isn't) then I'm OK with this going in as-is.

comment:10 by Carlton Gibson, 4 years ago

#31938 was a duplicate request similar functionality.

comment:11 by Mike Lissner, 2 years ago

Just to add a data points here, I spent a bunch of time today trying to figure out which key to delete to nuke a cached page and eventually gave up and just nuked a larger part of the cache in hopes of getting lucky (I did).

Having a helper util just to figure out what the cache key value is would be really nice.

Here's a summary of my failed research:

The reason I need this (and couldn't wait for the cache to just expire) is because I use the db cache to cache really big sitemaps that don't change very often. The cache is usually on the order of a couple of weeks.

comment:12 by Ahter Sönmez, 2 years ago

Owner: changed from nobody to Ahter Sönmez
Status: newassigned

comment:13 by Carlton Gibson, 22 months ago

#34271 was a duplicate with a suggestion to make the cache key just depend on the request path:

from pathlib import Path

from django.conf import settings
from django.core.cache import caches

default_cache = caches[settings.CACHE_MIDDLEWARE_ALIAS]
cache_key = ".".join(Path(request.path[1:]).parts)

if not (response := default_cache.get(cache_key)):
    response = view_func(request, *args, **kwargs)
    response.add_post_render_callback(
        lambda r: default_cache.set(cache_key, r, timeout)
    )

That would be easy to invalidate outside the request context.

Allowing/documenting easier make_key customisation might allow us to move this forward? 🤔

comment:14 by Petr Přikryl, 21 months ago

Cc: Petr Přikryl added

comment:15 by Ahter Sönmez, 21 months ago

I haven't had a chance to look at this since the sprint in DjangoCon Europe (nearly 6 months ago). As far as I remember, I already had "something" working.

In the next few days, let me check what was done, and share some progress here.

Last edited 21 months ago by Ahter Sönmez (previous) (diff)

comment:16 by Ahter Sönmez, 17 months ago

Here's a quick update on this:

I think this can be done. However, there are a few things to take into consideration.

  • Django's core caching code is quite old, 18+ years. I believe big part of it should be redesigned and refactored so it's easy to work on it and extend it in non-hacky ways. While this refactor is out of scope for this PR, I think it might be beneficial. I think this might be why this ticket is quite old.
  • Even when we solve this problem, due to the way cache keys are generated, this will be only supported by some cache backends (ie. Redis) which allow filtering by keys or part of keys.
  • There are better design patterns that would allow more intelligent caching in the projects/apps that use Django. Especially projects that use layered architecture. However, this is not applicable to Django as a library as it cannot be opinionated about how devs would like to structure their code in their apps.

I finally now have a chance to have a look at this again in DjangoCon Europe 2023 sprints. This time we've also paired with Marco Silva and we're tackling the issues together. I'm hoping to work on this more (given we're now two) and get something out as a PR soon.

comment:17 by Petr Dlouhý, 5 months ago

I would like to my use case to widen the perspective.
I am running a website with ~4k RPM and clearing cache for some view could result in timeouts due to cache hammering.
Multiple requests would bump into the cleared cache at the same time and it would result in too many DB requests which would slow down the single database significantly (the requests would take tens of seconds).

So clearing the cache even for single page/view is not an option for me.
The solution for me would be to directly inject new keys.

Version 0, edited 5 months ago by Petr Dlouhý (next)
Note: See TracTickets for help on using tickets.
Back to Top