Opened 3 months ago
Last modified 2 months ago
#35651 assigned Bug
RedisCacheClient does not reuse connections from the connection pool
Reported by: | gojuukaze | Owned by: | Ankit Jhunjhunwala |
---|---|---|---|
Component: | Core (Cache system) | Version: | 5.0 |
Severity: | Normal | Keywords: | asgi, async |
Cc: | Andrew Godwin, Carlton Gibson, Jon Janzen | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | yes |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
I'm using django.core.cache.backends.redis.RedisCache
as a backend for redis, and I recently observed that the number of Redis connections has been continuously increasing.
After debugging, I found that whenever the cache method is called, the RedisCacheClient
class is reinitialized, and in the __init__
function of RedisCacheClient
, the connection pool is set to empty.
This causes the _get_connection_pool
method of RedisCacheClient
to always create a new connection pool and a new connection instead of using the existing one.
class RedisCacheClient: def __init__( self, servers, serializer=None, pool_class=None, parser_class=None, **options, ): import redis self._lib = redis self._servers = servers self._pools = {} # << === set pools def _get_connection_pool(self, write): index = self._get_connection_pool_index(write) # # self._pools is is always empty. # if index not in self._pools: self._pools[index] = self._pool_class.from_url( self._servers[index], **self._pool_options, ) return self._pools[index]
One solution is to put _pools
outside of __init__
, for example:
class RedisCacheClient: # init pool _pools = {} def __init__( self, servers, serializer=None, pool_class=None, parser_class=None, **options, ): import redis self._lib = redis self._servers = servers ## self._pools={}
By the way, I am using Django 5.0.7 and running it in asynchronous mode.
Change History (15)
comment:1 by , 3 months ago
Description: | modified (diff) |
---|---|
Summary: | django redis cache not really using connection pooling → The number of Redis connections has been continuously increasing. |
follow-up: 4 comment:2 by , 3 months ago
Cc: | added |
---|
comment:3 by , 3 months ago
Resolution: | → needsinfo |
---|---|
Status: | new → closed |
comment:4 by , 3 months ago
Replying to Sarah Boyce:
Hi gojuukaze, thank you for raising this ticket
Would you be able to add some extra details to help replicate this? Perhaps your CACHE setting and a simple async view that touches the cache and instructions on how to see this ever increasing pool? Did you notice this after a Django upgrade?
You can use the code from this repository to reproduce the problem.
comment:5 by , 3 months ago
Resolution: | needsinfo |
---|---|
Status: | closed → new |
Summary: | The number of Redis connections has been continuously increasing. → RedisCacheClient does not reuse connections from the connection pool |
Triage Stage: | Unreviewed → Accepted |
Thank you for the test project gojuukaze!
comment:6 by , 3 months ago
Owner: | set to |
---|---|
Status: | new → assigned |
comment:8 by , 3 months ago
Replying to Ahmed Ibrahim:
I'm willing to help if needed
Ahmed, I haven't started on this ticket yet as a few things came up. I will un-assign myself so you can assign it to yourself!
comment:9 by , 3 months ago
Owner: | removed |
---|---|
Status: | assigned → new |
comment:10 by , 3 months ago
Owner: | set to |
---|---|
Status: | new → assigned |
comment:12 by , 3 months ago
Needs tests: | set |
---|
comment:13 by , 3 months ago
Needs tests: | unset |
---|
comment:14 by , 2 months ago
Cc: | added |
---|---|
Patch needs improvement: | set |
Copying my comment from the PR, because I don't think the approach suggested there is viable:
I haven't had the chance to fully dig in and see what/why it's happening but logging in BaseConnectionHandler.getitem() shows that we're not correctly caching the connection instance. The expected behaviour there is to create the instance on the first access, and then re-use it. That's not happening.
For some value of works, moving the pools to the class will work around it, but not for the correct reason, AFAICS.
The reproduce uses ASGI. Q: is this only happening in async code? I'm imagining so because the connections storage is a Local, so we'll be running into some thread sensitivity issue here: self._connections = Local(self.thread_critical)
More investigation needed.
comment:15 by , 2 months ago
Keywords: | asgi async added |
---|
Hi gojuukaze, thank you for raising this ticket
Would you be able to add some extra details to help replicate this? Perhaps your CACHE setting and a simple async view that touches the cache and instructions on how to see this ever increasing pool? Did you notice this after a Django upgrade?