Cache key not learnt for 304 responses — view must keep redoing work until "fresh" request comes in!

[Nathan Vander Wilt]

Discovered an interesting situation that's easy to hit in a development environment but might result in some wasted work in a production setting too. Consider this situation:

• I load a Django-rendered page in my browser. This browser keeps its own cache of the rendered response and its ETag.
• The Django cache on the serverside gets flushed somehow (e.g. if it's a LocMemCache and I'm using runserver this happens whenever I touch a file…)
• Now I (gently) referesh the page in my browser and it sends a request with If-None-Match and the ETag that it knows for the page.
• Since the cache was flushed the view has to render the page again. In this scenario the page hasn't changed, so after the render happens, the ETag matches — hooray! The browser gets a 304 Not Modified response back.

But here's the problem: the next time I gently refresh the browser (i.e. allow it to use its usual caching) the last step is repeated in whole! That is, the view can end up rendering the page AGAIN and AGAIN and each time the ETag matches in the end but Django does not seem to remember that.

As soon as a "fresh" request comes in, i.e. a more aggressive refresh from the browser that disables it's cache and yields a 200 response back, all subsequent "gentle refresh" responses then get a 304 *without* the view having to re-render.

I suspect that the culprit might be related to the logic in UpdateCacheMiddleware.process_response which will only call learn_cache_key when the response.status_code == 200?

[Nathan Vander Wilt]

Ah, I should note that my MIDDLEWARE ordering is:

[
    'UpdateCacheMiddleware',
    # … snip …
    'ConditionalGetMiddleware',
    # … snip …
    'FetchFromCacheMiddleware'
]

Perhaps that is incorrect? Is ConditionalGet intended to come after FetchFromCache or will that cause different problems?

[Nathan Vander Wilt]

Ah, this might be invalid. While the CacheMiddleware does handle 200 vs. 304 somewhat differently throughout its own logic, my impression is that the developer simply don't intend ConditionalGetMiddleware to work well and it is the real culprit here.

The <​https://docs.djangoproject.com/en/3.1/topics/conditional-view-processing/#comparison-with-middleware-conditional-processing> discussion referencing it warns:

It doesn’t save you from generating the response, which may be expensive.

and implies the goal of ± ETag handling is only much more modest such that:

[…] the amount of network traffic sent back to the clients will still be reduced if the view hasn’t changed.

Feel free to close as invalid if this is indeed the case.