Opened 3 weeks ago

Last modified 4 days ago

#36590 assigned Cleanup/optimization

Redis, Memcached, and Database cache backends lack specialized many operation implementations

Reported by: Brian Atkinson Owned by: H. White
Component: Core (Cache system) Version: 5.2
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The Redis cache backend has a specialized implementation of get_many and set_many which perform the get/set in one command to the backend. However the similar aget_many and aset_many methods are not specialized. This results in the performance of these async batch methods being significantly slower than they could be. We've seen significant performance improvements by adding the following methods to a sub-class of the built-in RedisCache class:

    async def aget_many(self, keys, version=None):
        return await sync_to_async(self.get_many)(keys, version=version)

    async def aset_many(self, data, timeout=DEFAULT_TIMEOUT, version=None):
        return await sync_to_async(self.set_many)(data, timeout=timeout, version=version)

These were the only ones we've overridden, but there might be a few others which could benefit from an improved implementation in a similar way. Happy to send a PR for a change similar to above for these missing specializations if desired.

Change History (7)

comment:1 by Simon Charette, 3 weeks ago

Component: UncategorizedCore (Cache system)
Summary: Redis cache lacks specialized aget_many and aset_many implementationsRedis, Memcached, and Database cache backends lack specialized many operation implementations
Triage Stage: UnreviewedAccepted
Type: UncategorizedCleanup/optimization

Memcached and database backends are also affected as well as delete_many

I wonder if we could potentially have BaseDatabaseCache.__init_subclass__ automatically add specialized aget_many, aset_many, adelete_many when one is declared by a subclass.

comment:2 by H. White, 2 weeks ago

Owner: set to H. White
Status: newassigned

comment:3 by H. White, 13 days ago

Has patch: set

comment:4 by Simon Charette, 13 days ago

Patch needs improvement: set

comment:5 by H. White, 12 days ago

Patch needs improvement: unset

comment:6 by Sarah Boyce, 4 days ago

Patch needs improvement: set

comment:7 by H. White, 4 days ago

Patch needs improvement: unset

I have revised the PR based on feedback.
https://github.com/django/django/pull/19840

Note: See TracTickets for help on using tickets.
Back to Top