SQLite threading tests are flaky when parallel test suite runs in forkserver mode

[Jacob Walls]

We have two tests often failing on GitHub Actions CI runs under the parallel test runner having to do with threading and sqlite in-memory databases.

• backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads
• servers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock

As of now, the failures are most common on the byte-compiled Django workflow, but we've at least seen the test_in_memory_database_lock failure on other workflows.

Tracebacks:

======================================================================
FAIL: test_database_sharing_in_threads (backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 669, in run
    self._callTestMethod(testMethod)
    
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 615, in _callTestMethod
    result = method()
    ^^^^^^^^^^^^^^^
  File "/home/runner/work/django/django/tests/backends/sqlite/tests.py", line 282, in test_database_sharing_in_threads
    self.assertEqual(Object.objects.count(), 2)
    ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual
    assertion_func(first, second, msg=msg)
    ^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual
    raise self.failureException(msg)
    ^^^^^^^^^^^
AssertionError: 1 != 2

----------------------------------------------------------------------

test_in_memory_database_lock (servers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock) failed:

    AssertionError('Unexpected error due to a database lock.')

Other times, the workflow deadlocks, so we don't know which test failed, which caused us to add timeout-minutes: 60 everywhere defensively in e48527f91d341c85a652499a5baaf725d36ae54f.

This failure started manifesting after we upgraded more CI jobs to Python 3.14, which defaults POSIX systems to the forkserver multiprocessing mode. See #36531.

I haven't been successful reproducing locally.

I have a low-confidence hypothesis that it might be something do with calls to setup_worker_connection inside _init_worker that occur during the middle of the test runs when there are resource contentions. Is it possible that a late worker init is clobbering some of these particular tests' setup where database connections are being overwritten to be the same (?)

[Natalia Bidart]

Thank you!

[Jacob Walls]

I haven't verified that this affects "spawn", so retitling.

[Jacob Walls]

I've seen this intermittently locally. Leaving aside the assertion failures for SQLite, we shouldn't have a hang in LiveServerTestCase when tests fail. You can engineer a hang like this, setting a miniscule timeout that will always raise:

diff --git a/django/test/testcases.py b/django/test/testcases.py
index 5f83612fe5..622b938dd6 100644
--- a/django/test/testcases.py
+++ b/django/test/testcases.py
@@ -1844,7 +1844,8 @@ class LiveServerTestCase(TransactionTestCase):
         cls.addClassCleanup(cls._terminate_thread)
 
         # Wait for the live server to be ready
-        cls.server_thread.is_ready.wait()
+        if not cls.server_thread.is_ready.wait(timeout=0.001):
+            raise Exception("Live server never became ready.")
         if cls.server_thread.error:
             raise cls.server_thread.error

Then when KeyboardInterrupting out of it, you get a stack trace from doClassCleanups, suggesting that the termination code is waiting forever, even though the live server never started:

  File "/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/unittest/suite.py", line 181, in _handleClassSetUp
    doClassCleanups()
    ~~~~~~~~~~~~~~~^^
  File "/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/unittest/case.py", line 720, in doClassCleanups
    function(*args, **kwargs)
    ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/django/django/test/testcases.py", line 1864, in _terminate_thread
    cls.server_thread.terminate()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/jwalls/django/django/test/testcases.py", line 1788, in terminate
    self.join()

Something like this fixes it:

django/test/testcases.py

@@ -1781 +1781 @@
         )
 
     def terminate(self):
-        if hasattr(self, "httpd"):
-            # Stop the WSGI server
-            self.httpd.shutdown()
-            self.httpd.server_close()
-        self.join()
+        if self.is_ready.is_set():
+            if hasattr(self, "httpd"):
+                # Stop the WSGI server
+                self.httpd.shutdown()
+                self.httpd.server_close()
+            self.join()

My theory is that the "live server never became ready" situation I simulated above is similar to the situation we're seeing on CI where a database lock entails a failure to start a live server thread.

---

Then for one of the underlying assertion failures, I don't know how I feel about masking a real problem, but we could probably reduce the chance of failing jobs by adjusting test_in_memory_database_lock() to use the other database instead of the default. It would still cover the code it's testing, but it would just have a much smaller chance of interacting poorly with other tests.

[Jacob Walls]

​PR

[Jacob Walls]

Replying to Jacob Walls:

we could probably reduce the chance of failing jobs by adjusting test_in_memory_database_lock() to use the other database instead of the default.

As Simon surmised on the above PR, this didn't help anything. (The PR still suffered from occasional failures in this test after trying this trick. I removed the speculative trick.)

[Jacob Walls <jacobtylerwalls@…>]

In 6c9ef62:

Refs #36770 -- Guarded against an endless wait in LiveServerThread.terminate().

terminate() shouldn't assume the main server was started. (A deadlock
from mishandling of in-memory SQLite databases may have occurred.)

[Jacob Walls <jacobtylerwalls@…>]

In 9c9a43b4:

Refs #36770 -- Preferred addCleanup() in live server tests.

[Jacob Walls <jacobtylerwalls@…>]

In afa026c:

Refs #36770 -- Skipped test_in_memory_database_lock().

Skip pending some investigation.

[Jacob Walls]

Per PR comment, attention is being placed on #27734 instead.

[Carlton Gibson]

Jacob pointed to a ​recent run that failed on Python 3.14t.

Same failure of FAIL: test_database_sharing_in_threads (backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads)

Higher up, that test outputs to stderr:

2026-07-16T08:41:39.8634599Z   File "/opt/hostedtoolcache/Python/3.14.6/x64-freethreaded/lib/python3.14t/threading.py", line 1082, in _bootstrap_inner
2026-07-16T08:41:39.8649851Z     self._context.run(self.run)
2026-07-16T08:41:39.8679847Z     ~~~~~~~~~~~~~~~~~^^^^^^^^^^
2026-07-16T08:41:39.8710359Z   File "/opt/hostedtoolcache/Python/3.14.6/x64-freethreaded/lib/python3.14t/threading.py", line 1024, in run
2026-07-16T08:41:39.8739891Z     self._target(*self._args, **self._kwargs)
2026-07-16T08:41:39.8769841Z     ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.8793248Z   File "/home/runner/work/django/django/tests/backends/sqlite/tests.py", line 277, in create_object
2026-07-16T08:41:39.8830003Z     Object.objects.create()
2026-07-16T08:41:39.8859919Z     ~~~~~~~~~~~~~~~~~~~~~^^
2026-07-16T08:41:39.8888118Z   File "/home/runner/work/django/django/django/db/models/manager.py", line 87, in manager_method
2026-07-16T08:41:39.8919938Z     return getattr(self.get_queryset(), name)(*args, **kwargs)
2026-07-16T08:41:39.8949855Z            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.8980349Z   File "/home/runner/work/django/django/django/db/models/query.py", line 736, in create
2026-07-16T08:41:39.9010356Z     obj.save(force_insert=True, using=self.db)
2026-07-16T08:41:39.9030085Z     ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.9061494Z   File "/home/runner/work/django/django/django/db/models/base.py", line 904, in save
2026-07-16T08:41:39.9100172Z     self.save_base(
2026-07-16T08:41:39.9150008Z     ~~~~~~~~~~~~~~^
2026-07-16T08:41:39.9159879Z         using=using,
2026-07-16T08:41:39.9160276Z         ^^^^^^^^^^^^
2026-07-16T08:41:39.9160622Z     ...<2 lines>...
2026-07-16T08:41:39.9160972Z         update_fields=update_fields,
2026-07-16T08:41:39.9161416Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.9161801Z     )
2026-07-16T08:41:39.9162071Z     ^
2026-07-16T08:41:39.9162978Z   File "/home/runner/work/django/django/django/db/models/base.py", line 996, in save_base
2026-07-16T08:41:39.9163709Z     updated = self._save_table(
2026-07-16T08:41:39.9164093Z         raw,
2026-07-16T08:41:39.9164398Z     ...<4 lines>...
2026-07-16T08:41:39.9164715Z         update_fields,
2026-07-16T08:41:39.9165033Z     )
2026-07-16T08:41:39.9165663Z   File "/home/runner/work/django/django/django/db/models/base.py", line 1199, in _save_table
2026-07-16T08:41:39.9166393Z     results = self._do_insert(
2026-07-16T08:41:39.9166918Z         cls._base_manager, using, insert_fields, returning_fields, raw
2026-07-16T08:41:39.9167457Z     )
2026-07-16T08:41:39.9168068Z   File "/home/runner/work/django/django/django/db/models/base.py", line 1251, in _do_insert
2026-07-16T08:41:39.9169215Z     return manager._insert(
2026-07-16T08:41:39.9169714Z            ~~~~~~~~~~~~~~~^
2026-07-16T08:41:39.9170047Z         [self],
2026-07-16T08:41:39.9170344Z         ^^^^^^^
2026-07-16T08:41:39.9170640Z     ...<3 lines>...
2026-07-16T08:41:39.9170948Z         raw=raw,
2026-07-16T08:41:39.9171242Z         ^^^^^^^^
2026-07-16T08:41:39.9171524Z     )
2026-07-16T08:41:39.9171790Z     ^
2026-07-16T08:41:39.9172430Z   File "/home/runner/work/django/django/django/db/models/manager.py", line 87, in manager_method
2026-07-16T08:41:39.9173276Z     return getattr(self.get_queryset(), name)(*args, **kwargs)
2026-07-16T08:41:39.9173844Z            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.9174619Z   File "/home/runner/work/django/django/django/db/models/query.py", line 2141, in _insert
2026-07-16T08:41:39.9175611Z     return query.get_compiler(using=using).execute_sql(returning_fields)
2026-07-16T08:41:39.9176267Z            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.9177022Z   File "/home/runner/work/django/django/django/db/models/sql/compiler.py", line 1936, in execute_sql
2026-07-16T08:41:39.9177772Z     cursor.execute(sql, params)
2026-07-16T08:41:39.9178120Z     ~~~~~~~~~~~~~~^^^^^^^^^^^^^
2026-07-16T08:41:39.9178772Z   File "/home/runner/work/django/django/django/db/backends/utils.py", line 79, in execute
2026-07-16T08:41:39.9179661Z     return self._execute_with_wrappers(
2026-07-16T08:41:39.9180337Z            ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
2026-07-16T08:41:39.9180805Z         sql, params, many=False, executor=self._execute
2026-07-16T08:41:39.9181314Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.9181715Z     )
2026-07-16T08:41:39.9181980Z     ^
2026-07-16T08:41:39.9182689Z   File "/home/runner/work/django/django/django/db/backends/utils.py", line 92, in _execute_with_wrappers
2026-07-16T08:41:39.9183519Z     return executor(sql, params, many, context)
2026-07-16T08:41:39.9184440Z   File "/home/runner/work/django/django/django/db/backends/utils.py", line 100, in _execute
2026-07-16T08:41:39.9185165Z     with self.db.wrap_database_errors:
2026-07-16T08:41:39.9185586Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-16T08:41:39.9186239Z   File "/home/runner/work/django/django/django/db/utils.py", line 94, in __exit__
2026-07-16T08:41:39.9187049Z     raise dj_exc_value.with_traceback(traceback) from exc_value
2026-07-16T08:41:39.9187876Z   File "/home/runner/work/django/django/django/db/backends/utils.py", line 105, in _execute
2026-07-16T08:41:39.9188612Z     return self.cursor.execute(sql, params)
2026-07-16T08:41:39.9189062Z            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
2026-07-16T08:41:39.9220225Z   File "/home/runner/work/django/django/django/db/backends/sqlite3/base.py", line 359, in execute
2026-07-16T08:41:39.9221110Z     return super().execute(query, params)
2026-07-16T08:41:39.9221563Z            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
2026-07-16T08:41:39.9222138Z django.db.utils.OperationalError: no such table: backends_object
2026-07-16T08:41:39.9450775Z test_database_sharing_in_threads (backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads) ... FAIL

That looks like (somehow) the worker thread resolved a non-existent database name, and so when connecting to it (in memory, so making a new one) there's no schema in place. 🤔

Is it possible that a late worker init is clobbering some of these particular tests' setup where database connections are being overwritten to be the same (?)

Which is that, no?

Not a fix but, can we check that the DB NAME in the worker matches what we're expecting from the main thread? (That would at least show that settings are getting mangled if not... 🤔)