Opened 4 years ago

Closed 4 years ago

#33252 closed Bug (fixed)

CacheMiddleware and FetchFromCacheMiddleware are not thread safe.

Reported by: Iuri de Silvio Owned by: Iuri de Silvio
Component: Core (Cache system) Version: 3.2
Severity: Normal Keywords:
Cc: Nick Pope Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

CacheMiddleware persist self.cache = caches[cache_alias] on startup and it is not thread safe. ​https://github.com/django/django/blob/main/django/middleware/cache.py#L186

I found that after some production errors with pylibmc and uwsgi threaded. Created a small project to reproduce it. Nothing fancy, just pylibmc cache and a @cache_page cached view. It fails even with development server, with concurrent requests.

Traceback (most recent call last):
  File "versions/pylibmcbug/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "versions/pylibmcbug/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "versions/pylibmcbug/lib/python3.9/site-packages/django/utils/decorators.py", line 122, in _wrapped_view
    result = middleware.process_request(request)
  File "versions/pylibmcbug/lib/python3.9/site-packages/django/middleware/cache.py", line 145, in process_request
    cache_key = get_cache_key(request, self.key_prefix, 'GET', cache=self.cache)
  File "versions/pylibmcbug/lib/python3.9/site-packages/django/utils/cache.py", line 362, in get_cache_key
    headerlist = cache.get(cache_key)
  File "versions/pylibmcbug/lib/python3.9/site-packages/django/core/cache/backends/memcached.py", line 77, in get
    return self._cache.get(key, default)
pylibmc.ConnectionError: error 3 from memcached_get(:1:views.decorators.cache.cache_): (0x7f290400bd60) FAILURE, poll() returned a value that was not dealt with,  host: localhost:11211 -> libmemcached/io.cc:254

Looking for git history, it is this way since 2010. ​https://github.com/django/django/commit/673e6fc7fb243ed44841b9969d26a161c25733b3

Change History (8)

comment:1 by Iuri de Silvio, 4 years ago

Owner: changed from nobody to Iuri de Silvio
Status: new → assigned

PR ​https://github.com/django/django/pull/15044

comment:2 by Iuri de Silvio, 4 years ago

Possibly related to #33092.

comment:3 by Iuri de Silvio, 4 years ago

Has patch: set

comment:4 by Iuri de Silvio, 4 years ago

Component: Uncategorized → Core (Cache system)

comment:5 by Mariusz Felisiak, 4 years ago

Cc: Nick Pope added
Summary: CacheMiddleware is not thread safe → CacheMiddleware and FetchFromCacheMiddleware are not thread safe.
Triage Stage: Unreviewed → Accepted
Type: Uncategorized → Bug

Thanks for the report!

comment:6 by Mariusz Felisiak, 4 years ago

Triage Stage: Accepted → Ready for checkin

comment:7 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In 0c05c18:

Refs #33252 -- Used @override_settings in BaseCacheTests.test_cache_write_unpicklable_object()

comment:8 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

Resolution: → fixed
Status: assigned → closed

In 3ff7b15:

Fixed #33252 -- Made cache middlewares thread-safe.

Note: See TracTickets for help on using tickets.
Back to Top