RedisCacheClient does not reuse connections from the connection pool

[gojuukaze]

I'm using django.core.cache.backends.redis.RedisCache as a backend for redis, and I recently observed that the number of Redis connections has been continuously increasing.

After debugging, I found that whenever the cache method is called, the RedisCacheClient class is reinitialized, and in the __init__ function of RedisCacheClient, the connection pool is set to empty.

This causes the _get_connection_pool method of RedisCacheClient to always create a new connection pool and a new connection instead of using the existing one.

class RedisCacheClient:
    def __init__(
        self,
        servers,
        serializer=None,
        pool_class=None,
        parser_class=None,
        **options,
    ):
        import redis

        self._lib = redis
        self._servers = servers
        self._pools = {} # << === set pools

    def _get_connection_pool(self, write):
        index = self._get_connection_pool_index(write)

       #
       # self._pools is is always empty.
       #

        if index not in self._pools:
            self._pools[index] = self._pool_class.from_url(
                self._servers[index],
                **self._pool_options,
            )
        return self._pools[index]

One solution is to put _pools outside of __init__ , for example：

class RedisCacheClient:

    # init pool
    _pools = {} 

    def __init__(
        self,
        servers,
        serializer=None,
        pool_class=None,
        parser_class=None,
        **options,
    ):
        import redis

        self._lib = redis
        self._servers = servers
        ## self._pools={}

By the way, I am using Django 5.0.7 and running it in asynchronous mode.

[Sarah Boyce]

Hi gojuukaze, thank you for raising this ticket
Would you be able to add some extra details to help replicate this? Perhaps your CACHE setting and a simple async view that touches the cache and instructions on how to see this ever increasing pool? Did you notice this after a Django upgrade?

[gojuukaze]

Replying to Sarah Boyce:

Hi gojuukaze, thank you for raising this ticket
Would you be able to add some extra details to help replicate this? Perhaps your CACHE setting and a simple async view that touches the cache and instructions on how to see this ever increasing pool? Did you notice this after a Django upgrade?

You can use the code from this repository to reproduce the problem.

​https://github.com/gojuukaze/django-35651

[Sarah Boyce]

Thank you for the test project gojuukaze!

[Ahmed Ibrahim]

I'm willing to help if needed

[Peter Williams]

Replying to Ahmed Ibrahim:

I'm willing to help if needed

Ahmed, I haven't started on this ticket yet as a few things came up. I will un-assign myself so you can assign it to yourself!

[Ankit Jhunjhunwala]

​https://github.com/django/django/pull/18459

[Carlton Gibson]

Copying ​my comment from the PR, because I don't think the approach suggested there is viable:

I haven't had the chance to fully dig in and see what/why it's happening but logging in BaseConnectionHandler.getitem() shows that we're not correctly caching the connection instance. The expected behaviour there is to create the instance on the first access, and then re-use it. That's not happening.

For some value of works, moving the pools to the class will work around it, but not for the correct reason, AFAICS.

The reproduce uses ASGI. Q: is this only happening in async code? I'm imagining so because the connections storage is a Local, so we'll be running into some thread sensitivity issue here: self._connections = Local(self.thread_critical)

More investigation needed.