Opened 82 minutes ago
#36770 new Cleanup/optimization
SQLite threading tests are flaky when parallel test suite runs in forkserver/spawn
| Reported by: | Jacob Walls | Owned by: | |
|---|---|---|---|
| Component: | Testing framework | Version: | 5.2 |
| Severity: | Normal | Keywords: | 3.14, forkserver, spawn, parallel |
| Cc: | Triage Stage: | Unreviewed | |
| Has patch: | no | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
We have two tests often failing on GitHub Actions CI runs under the parallel test runner having to do with threading and sqlite in-memory databases.
backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threadsservers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock
As of now, the failures are most common on the byte-compiled Django workflow, but we've at least seen the test_in_memory_database_lock failure on other workflows.
Tracebacks:
====================================================================== FAIL: test_database_sharing_in_threads (backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 58, in testPartExecutor yield File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 669, in run self._callTestMethod(testMethod) File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 615, in _callTestMethod result = method() ^^^^^^^^^^^^^^^ File "/home/runner/work/django/django/tests/backends/sqlite/tests.py", line 282, in test_database_sharing_in_threads self.assertEqual(Object.objects.count(), 2) ^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual assertion_func(first, second, msg=msg) ^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual raise self.failureException(msg) ^^^^^^^^^^^ AssertionError: 1 != 2 ----------------------------------------------------------------------
test_in_memory_database_lock (servers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock) failed: AssertionError('Unexpected error due to a database lock.')
Other times, the workflow deadlocks, so we don't know which test failed, which caused us to add timeout-minutes: 60 everywhere defensively in e48527f91d341c85a652499a5baaf725d36ae54f.
This failure started manifesting after we upgraded more CI jobs to Python 3.14, which defaults POSIX systems to the forkserver multiprocessing mode. See #36531.
I haven't been successful reproducing locally.
I have a low-confidence hypothesis that it might be something do with calls to setup_worker_connection inside _init_worker that occur during the middle of the test runs when there are resource contentions. Is it possible that a late worker init is clobbering some of these particular tests' setup where database connections are being overwritten to be the same (?)