Running tests on SQLite with --parallel (using spawn) does not respect DATABASES["TEST"]["NAME"]

[Sage Abdullah]

When running unit tests on SQLite on macOS with the --parallel ​flag and the DATABASES["TEST"]["NAME"] ​setting set (e.g. to mytests.db), the tests fail to run because SQLite tries to connect to databases named default_*.sqlite3 instead of the cloned mytests_*.db.

You can replicate this with the Django test suite itself, e.g. with the following change:

tests/test_sqlite.py

@@ -29 +29 @@
 ]
 
 USE_TZ = False
+
+import os
+
+DATABASE_NAME = os.getenv("DATABASE_NAME")
+if DATABASE_NAME:
+    DATABASES["default"]["TEST"] = {"NAME": DATABASE_NAME}

and then run a subset of the tests, e.g.

DATABASE_NAME=django.db ./runtests.py --verbosity=2 --parallel=2 -- model_fields

You'll get a bunch of errors like the following:

Traceback (most recent call last):
  File "/../python/3.14.2/lib/python3.14/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/../python/3.14.2/lib/python3.14/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/../python/3.14.2/lib/python3.14/multiprocessing/pool.py", line 109, in worker
    initializer(*initargs)
    ~~~~~~~~~~~^^^^^^^^^^^
  File "/django/django/test/runner.py", line 479, in _safe_init_worker
    init_worker(counter, *args, **kwargs)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/django/django/test/runner.py", line 464, in _init_worker
    connection.creation.setup_worker_connection(_worker_id)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/django/django/db/backends/sqlite3/creation.py", line 145, in setup_worker_connection
    source_db = self.connection.Database.connect(
        f"file:{alias}_{_worker_id}.sqlite3?mode=ro", uri=True
    )
sqlite3.OperationalError: unable to open database file

but you can see that Django successfully created/destroyed the test databases with the correct names:

Cloning test database for alias 'default' ('django.db')...
Cloning test database for alias 'default' ('django.db')...

# test results go here

Destroying test database for alias 'default' ('django_1.db')...
Destroying test database for alias 'default' ('django_2.db')...
Destroying test database for alias 'default' ('django.db')...

This also affects a multi-database setup if you set the ["TEST"]["NAME"] for that database. With the above example, if you run the multiple_database test suite (instead of model_fields), you won't see any errors for the other database; as Django will create and connect to the fallback test name based on the alias, i.e. other_*.sqlite3. However, if you also add DATABASES["other"]["TEST"] = {"NAME": "other_" + DATABASE_NAME} to the settings changes in the above example, the same error now also happens for the other database.

This prevents the use of custom names for running tests with --parallel > 1 (or auto), which can be particularly useful when you want to use --keepdb with different names. With --parallel=1, this issue does not occur.

I believe it is because despite the cloning was done correctly using the given test database file name in ​get_test_db_clone_settings, the logic in ​setup_worker_connection does not use the clone db names (it always uses {alias}_{_worker_id}.sqlite3 instead) when copying them over to the in-memory database.

FWIW, I encountered this while trying to run ​Wagtail's test suite in parallel with --keepdb. I can give a smaller reproducible example if necessary, but I think the example with Django's own test suite can be a good example too.

[Jacob Walls]

Thanks Sage, I suspected this was not handled but hadn't yet verified. Would you like to submit a PR?

the logic in ​setup_worker_connection does not use the clone db names (it always uses {alias}_{_worker_id}.sqlite3 instead) when copying them over to the in-memory database.

Related: I've noticed we have the same OperationalError when a parallel test worker dies. In that case, the worker_id increments past the number of databases, and a connection is requested for a nonexistent database. At a glance, it looks like your suggested patch fixes that as well.

[Sriniketh99]

Hi!
I'm interested in working on this ticket.
I'm still learning about Django internals, but I'd like to begin by trying to reproduce this problem myself and see how the test runner is setting up the SQLite databases with the --parallel option.
My approach would be to try to understand how the test database names are being determined and passed down to the worker processes, and then attempt to solve the problem.
I'll post back here as I make progress and may have questions if I get stuck.

[Sage Abdullah]

Replying to Jacob Walls:

Thanks Sage, I suspected this was not handled but hadn't yet verified. Would you like to submit a PR?

Yep, I'll submit a PR soon. Just trying to find the most appropriate way to test it.

Related: I've noticed we have the same OperationalError when a parallel test worker dies. In that case, the worker_id increments past the number of databases, and a connection is requested for a nonexistent database. At a glance, it looks like your suggested patch fixes that as well.

Awesome!

Replying to Sriniketh99:

Hi!
I'm interested in working on this ticket.
I'm still learning about Django internals, but I'd like to begin by trying to reproduce this problem myself and see how the test runner is setting up the SQLite databases with the --parallel option.
My approach would be to try to understand how the test database names are being determined and passed down to the worker processes, and then attempt to solve the problem.
I'll post back here as I make progress and may have questions if I get stuck.

Sorry, I should've assigned the ticket to myself. I already made the patch, so I just need to submit the PR. I'd recommend looking for a different ticket to work on.

[Sage Abdullah]

​PR

[GitHub <noreply@…>]

In b415cef:

Fixed #36946 -- Respected test database name when running tests in parallel on SQLite.

The "spawn" and "forkserver" multiprocessing modes were affected.