Opened 2 years ago
Last modified 2 years ago
#32831 new Cleanup/optimization
Some cache tests are flaky (e.g. cache.tests.BaseMemcachedTests.test_touch)
Reported by: | Chris Jerdonek | Owned by: | |
---|---|---|---|
Component: | Core (Cache system) | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Accepted | |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
The following tests occasionally fail in CI:
cache.tests.PyLibMCCacheTests.test_touch
cache.tests.PyMemcacheCacheTests.test_touch
Traceback (most recent call last): File "/home/jenkins/workspace/pull-requests-bionic/database/sqlite3/label/bionic-pr/python/python3.8/tests/cache/tests.py", line 481, in test_touch self.assertIs(cache.touch('expire1'), True) AssertionError: False is not True
The line that fails is here: https://github.com/django/django/blob/ed3af3ff4b3cfb72de598f1b39a1028ba3826efb/tests/cache/tests.py#L481
Both test classes inherit from BaseMemcachedTests
, which inherits from BaseCacheTests
.
Change History (11)
comment:1 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:2 Changed 2 years ago by
Triage Stage: | Unreviewed → Accepted |
---|
comment:3 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:4 Changed 2 years ago by
Could this be due to different concrete subclasses of BaseMemcachedTests
stepping on each other during a parallel run? I ask because different test classes are still using the same cache key.
comment:5 Changed 2 years ago by
Another cache-related flaky test is: cache.tests.CacheMiddlewareTest.test_cache_page_timeout
on this line.
Traceback (most recent call last): File "C:\Jenkins\workspace\pull-requests-windows\database\sqlite3\label\windows\python\Python39\tests\cache\tests.py", line 2449, in test_cache_page_timeout self.assertEqual( AssertionError: b'Hello World 1' != b'Hello World 2'
comment:6 Changed 2 years ago by
A possible pattern for fixing issues like this would be to keep retrying the assertion in a while
loop, with a small sleep in between each attempt, and a relatively large cumulative amount of time before giving up. That would make it as pass as quickly as possible, without waiting longer than necessary. I'm thinking of something like--
def assert_with_retries(self, assert_func, *args, interval_time=0.1, total_time=10, **kwargs): start = time.time() while True: try: assert_func(*args, **kwargs) except Exception as exc: if time.time() - start > total_time: raise RuntimeError(f'Total retry time exceeded.') from exc else: break time.sleep(interval_time)
comment:7 Changed 2 years ago by
Something else that occurred to me is that it could only help to start each test by asserting that each key being used isn't set, and to end each test by clearing the key in a finally block. The former would help to show if a test isolation issue is at fault.
comment:8 Changed 2 years ago by
Summary: | Flaky test in CI: cache.tests.BaseMemcachedTests.test_touch → Some cache tests are flaky (e.g. cache.tests.BaseMemcachedTests.test_touch) |
---|
comment:9 Changed 2 years ago by
Owner: | changed from nobody to Chris Jerdonek |
---|---|
Status: | new → assigned |
comment:10 Changed 2 years ago by
Another flaky test failure: cache.tests.MemcachedCacheTests.test_forever_timeout()
at this line-- https://github.com/django/django/blob/1697098795707bd39501d9ceddd3d9be93f8d549/tests/cache/tests.py#L606
(again with pull-requests-bionic/database=spatialite,label=bionic-pr,python=python3.8
).
comment:11 Changed 2 years ago by
Owner: | Chris Jerdonek deleted |
---|---|
Status: | assigned → new |
Yes these tests are flaky (and probably 2-3 more cache tests). We tried to fix them in the past but with no success (see PR).