Opened 7 months ago

Closed 7 months ago

Last modified 7 months ago

#32747 closed Bug (fixed)

CacheHandler initialize unused caches.

Reported by: Alexander Ebral Owned by: Mariusz Felisiak
Component: Core (Cache system) Version: 3.2
Severity: Release blocker Keywords:
Cc: Florian Apolloner Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

After the commit: https://github.com/django/django/commit/98e05ccde440cc9b768952cc10bc8285f4924e1f
logic of the method "all" from CacheHandler class was changed.

Before:

    def all(self):
        return getattr(self._caches, 'caches', {}).values()

This method returned connections that were created in __getitem__

Now:

    def all(self):
        return [self[alias] for alias in self]

Connections return for all "CACHES" from settings.py (in case of absence - they are forcibly created in self[alias])

Which version of this method seems to be right?

In my case this unnecessary mass initialization of custom diskcache-classes leads to io-lags.

Snippet that helped me:

import django.core.cache


def cache_getitem(self, alias, exists_only=False):
    try:
        return getattr(self._connections, alias)
    except AttributeError:
        if alias not in self.settings:
            raise self.exception_class(f"The connection '{alias}' doesn't exist.")
        if exists_only:
            return
    conn = self.create_connection(alias)
    setattr(self._connections, alias, conn)
    return conn


def cache_all(self):
    connections = [self.__getitem__(alias, exists_only=True) for alias in self]
    return [conn for conn in connections if conn is not None]


django.core.cache.CacheHandler.all = cache_all
django.core.cache.CacheHandler.__getitem__ = cache_getitem

Change History (5)

comment:1 Changed 7 months ago by Mariusz Felisiak

Cc: Florian Apolloner added
Severity: NormalRelease blocker
Summary: CacheHandler logic issueCacheHandler initialize unused caches.
Triage Stage: UnreviewedAccepted

Thanks for the report. I agree this can cause a performance regression, we shouldn't initialize unused caches unnecessarily. As far as I'm aware we can restore the previous behavior with:

diff --git a/django/core/cache/__init__.py b/django/core/cache/__init__.py
index 05ef3897d0..c008ed1125 100644
--- a/django/core/cache/__init__.py
+++ b/django/core/cache/__init__.py
@@ -43,6 +43,8 @@ class CacheHandler(BaseConnectionHandler):
             ) from e
         return backend_cls(location, params)
 
+    def all(self):
+        return [self[alias] for alias in self if hasattr(self._connections, alias)]
 
 caches = CacheHandler()
 

Regression in 98e05ccde440cc9b768952cc10bc8285f4924e1f.

comment:2 Changed 7 months ago by Mariusz Felisiak

Owner: changed from nobody to Mariusz Felisiak
Status: newassigned

comment:3 Changed 7 months ago by Mariusz Felisiak

Has patch: set

comment:4 Changed 7 months ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In 958cdf65:

Fixed #32747 -- Prevented initialization of unused caches.

Thanks Alexander Ebral for the report.

Regression in 98e05ccde440cc9b768952cc10bc8285f4924e1f.

comment:5 Changed 7 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In 41e2aa7:

[3.2.x] Fixed #32747 -- Prevented initialization of unused caches.

Thanks Alexander Ebral for the report.

Regression in 98e05ccde440cc9b768952cc10bc8285f4924e1f.

Backport of 958cdf65ae90d26236d1815bbba804729595ec7a from main

Note: See TracTickets for help on using tickets.
Back to Top