Opened 3 weeks ago

Last modified 5 days ago

#36770 assigned Cleanup/optimization

SQLite threading tests are flaky when parallel test suite runs in forkserver mode

Reported by: Jacob Walls Owned by: Kundan Yadav
Component: Testing framework Version: 5.2
Severity: Normal Keywords: 3.14, forkserver, parallel
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

We have two tests often failing on GitHub Actions CI runs under the parallel test runner having to do with threading and sqlite in-memory databases.

  • backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads
  • servers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock

As of now, the failures are most common on the byte-compiled Django workflow, but we've at least seen the test_in_memory_database_lock failure on other workflows.

Tracebacks:

======================================================================
FAIL: test_database_sharing_in_threads (backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 669, in run
    self._callTestMethod(testMethod)
    
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 615, in _callTestMethod
    result = method()
    ^^^^^^^^^^^^^^^
  File "/home/runner/work/django/django/tests/backends/sqlite/tests.py", line 282, in test_database_sharing_in_threads
    self.assertEqual(Object.objects.count(), 2)
    ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual
    assertion_func(first, second, msg=msg)
    ^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual
    raise self.failureException(msg)
    ^^^^^^^^^^^
AssertionError: 1 != 2

----------------------------------------------------------------------
test_in_memory_database_lock (servers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock) failed:

    AssertionError('Unexpected error due to a database lock.')

Other times, the workflow deadlocks, so we don't know which test failed, which caused us to add timeout-minutes: 60 everywhere defensively in e48527f91d341c85a652499a5baaf725d36ae54f.

This failure started manifesting after we upgraded more CI jobs to Python 3.14, which defaults POSIX systems to the forkserver multiprocessing mode. See #36531.

I haven't been successful reproducing locally.

I have a low-confidence hypothesis that it might be something do with calls to setup_worker_connection inside _init_worker that occur during the middle of the test runs when there are resource contentions. Is it possible that a late worker init is clobbering some of these particular tests' setup where database connections are being overwritten to be the same (?)

Change History (3)

comment:1 by Natalia Bidart, 3 weeks ago

Triage Stage: UnreviewedAccepted

Thank you!

comment:2 by Kundan Yadav, 9 days ago

Owner: set to Kundan Yadav
Status: newassigned

comment:3 by Jacob Walls, 5 days ago

Keywords: spawn removed
Summary: SQLite threading tests are flaky when parallel test suite runs in forkserver/spawnSQLite threading tests are flaky when parallel test suite runs in forkserver mode

I haven't verified that this affects "spawn", so retitling.

Note: See TracTickets for help on using tickets.
Back to Top