Missing parent directory for SQLite test database path not created

[Damian Posener]

How to replicate:

1. Have a DATABASES configuration with one or more SQLite DBs in directories, like this:

   DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.sqlite3',
           'NAME': BASE_DIR / 'db' / 'default' / 'db.sqlite3',
       },
       'other': {
           'ENGINE': 'django.db.backends.sqlite3',
           'NAME': BASE_DIR / 'db' / 'other' / 'db.sqlite3',
       }
   }
   

2. Run tests with both the --parallel=auto and --keepdb flags, i.e. python manage.py test --keepdb --parallel=auto

Expected behaviour: .sqlite3 files should be kept in db/default and db/other directories

Actual behaviour: default_X.sqlite3 and other_X.sqlite3 files are saved in the root directory instead

Why is this a problem? When running multiple test suites at once, this leads to filename collision if two test suites are using the same database name(s). It is impossible to run two Django test suites in parallel from the same directory without encountering this issue.

Setting DATABASES['<alias>']['TEST']['NAME'] does not seem to solve this either.

I've put together an example project to demonstrate the issue. Simply run python manage.py test --keepdb --parallel=auto from the base directory.

Happy to answer any other questions. :)

[Abhishek Srivastava]

Hi 👋 — I’m trying to reproduce this bug to help confirm .

Here’s what I did so far:

1. Created a test project with the DATABASES config exactly as described.
2. Ran migrate — the .sqlite3 file was created correctly in db/default/.
3. Added a simple TestCase with multiple dummy tests.
4. Ran python manage.py test --keepdb --parallel=auto.

Environment:

• Django version: 6.0.dev20250612075530 (editable install)
• Python version: 3.12.11
• SQLite version: 3.43.2

I only see db/default/db.sqlite3 — no additional default_1.sqlite3 or other_1.sqlite3 files appear in the project root.

Am I missing any step to trigger the parallel test DB creation?
Any suggestions are appreciated — thanks!

[Simon Charette]

Why is this a problem? When running multiple test suites at once, this leads to filename collision if two test suites are using the same database name(s). It is impossible to run two Django test suites in parallel from the same directory without encountering this issue.

I don't think that's something Django's test machinery claims to supports by the way, I can't think of multiple things that would break running multiple test suites against the same checkout of a projects.

Am I missing any step to trigger the parallel test DB creation?

You need to have at least two TestCase otherwise ​these is nothing to parallelize.

[Damian Posener]

Replying to abhishek1999:

Hi 👋 — I’m trying to reproduce this bug to help confirm .

Here’s what I did so far:

1. Created a test project with the DATABASES config exactly as described.
2. Ran migrate — the .sqlite3 file was created correctly in db/default/.
3. Added a simple TestCase with multiple dummy tests.
4. Ran python manage.py test --keepdb --parallel=auto.

Environment:

• Django version: 6.0.dev20250612075530 (editable install)
• Python version: 3.12.11
• SQLite version: 3.43.2

I only see db/default/db.sqlite3 — no additional default_1.sqlite3 or other_1.sqlite3 files appear in the project root.

Am I missing any step to trigger the parallel test DB creation?
Any suggestions are appreciated — thanks!

There is a demo project attached to this ticket as a zip file. If you extract that and run tests you can replicate the issue hopefully.

[Damian Posener]

Replying to Simon Charette:

I don't think that's something Django's test machinery claims to supports by the way, I can't think of multiple things that would break running multiple test suites against the same checkout of a projects.

We are using Django as a CMS for 14 different websites, and the nature of that beast is a lot of separate apps, each with their own separate test suites. Being able to run app tests in parallel saves us a bunch of time, but this database thing is a bit of a thorn in our side.

I think there's also some (probably minor) security risks around Django simply dumping it's databases copies in the current working directory. For one the directory might not actually be writable. If it is writable, it may contain other .sqlite3 files that aren't meant to be overwritten (Django does not seem to check whether the DBs are its own and will happily overwrite existing ones).

Personally I'd be happy if there was at least a way to specify where Django should store the files, but there seems to be no other option than to use the current working dir.

[Simon Charette]

Personally I'd be happy if there was at least a way to specify where Django should store the files, but there seems to be no other option than to use the current working dir.

Have you tried using the explicit TEST.NAME ​setting?

I missed that part

Setting DATABASES['<alias>']['TEST']['NAME'] does not seem to solve this either.

I think we can accept on the basis that TEST.NAME should work.

[Abhishek Srivastava]

You need to have at least two TestCase otherwise ​these is nothing to parallelize.

Thanks, I was able to reproduce the issue.

[Abhishek Srivastava]

I tried this:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": BASE_DIR / "db" / "default" / "db.sqlite3",
        "TEST": {
            "NAME": BASE_DIR / "db" / "default" / "test_db.sqlite3",
        },
    },
    "other": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": BASE_DIR / "db" / "other" / "db.sqlite3",
        "TEST": {
            "NAME": BASE_DIR / "db" / "other" / "test_db.sqlite3",
        },
    },
}

I added ['TEST']['NAME'] to keep test DB files in separate folders.
I ran tests with:

python manage.py test --parallel=auto --keepdb

Observations:

• Django tries to create test DB copies with suffixes (_1, _2, etc.) for each worker process
• The base .sqlite3 test files were created in a different folder than the project root, as configured.
• But when running in parallel, Django does not handle the folder part when adding suffixes . It gives following error

  sqlite3.OperationalError: unable to open database file
  

[Simon Charette]

Damian, do you have a requirement that parallel test databases are backed by actual files instead of being in memory per worker process which should ensure isolation?

What I mean is that if you want to avoid collisions until this gets fixed you could simply set DATABASES['<alias>']['TEST']['NAME'] = ':memory:' which is a common setup to speed up tests and ensure isolation.

[Damian Posener]

Replying to Simon Charette:

Damian, do you have a requirement that parallel test databases are backed by actual files instead of being in memory per worker process which should ensure isolation?

What I mean is that if you want to avoid collisions until this gets fixed you could simply set DATABASES['<alias>']['TEST']['NAME'] = ':memory:' which is a common setup to speed up tests and ensure isolation.

That would rather defeat the point of using --keepdb, don't you think? :D We do run with in-memory DBs for a few applications, but that's an inefficient choice when testing apps that have a load of migrations in them.

[Gangadhar Yadav]

Replying to Gangadhar Yadav:
Working on this and have a near-final fix locally. The change ensures SQLite parallel test databases are created in the same directories as the configured database files (not the project root), preventing filename collisions. I’ve verified with multiple DB aliases and

--keepdb/--parallel=auto

preparing a PR with tests and details shortly. Expect a draft PR within the next 1–2 days.

[Gangadhar Yadav]

PR: ​https://github.com/django/django/pull/19864

This implements the fix and adds a regression test in
tests/backends/sqlite/test_creation.py::SQLiteParallelCloneTests.

[Becky Smith]

I think this bug is simpler than initially reported; the issue is when a sqlite DB attempts to create a file on disk in a directory that doesn't yet exist.

Just running manage.py runserver with these settings hits the same unable to open database file error (where unknown_dir hasn't been created). If the dir exists but the db file does not, it creates the file in the empty directory as expected.

DATABASES = {
   'default': {
       'ENGINE': 'django.db.backends.sqlite3',
       'NAME': BASE_DIR / 'unknown_dir' / 'db.sqlite3',
   }
}

Running tests passes, because the sqlite backend by default uses an in-memory db and doesn't write to file.

However, with these settings, running the tests fails (any single test), because using the explicit TEST setting attempts to create the file on disk.

DATABASES = {
   'default': {
       'ENGINE': 'django.db.backends.sqlite3',
       'NAME': {BASE_DIR / 'unknown_dir' / 'db.sqlite3',
       'TEST': {'NAME': BASE_DIR /  'unknown_dir' / 'testdb.sqlite3'}
   }
}

Running tests with --keepdb unexpectedly creates the test db files in the base directory (ignoring the full path from the settings) if no TEST setting is provided. If there is TEST setting, it puts the test files in the expected directory.

As far as I can tell, running the tests with --parallel auto and --keepdb works fine, without filename collisions. Each worker creates a db file with the db name and a numeric suffix (default_1.sqlite3, other_1.sqlite). If there are no TEST db settings specified, these are all created at the base directory level, but they don't conflict. If there are TEST db settings specified, they are created in the specified path. As long as this path exists, the files are created with numeric suffixes in those directories, again no conflicts.

[Becky Smith]

I misread the original report in this ticket, which was that db names collide when running multiple test suites at the same time while also running with --keepdb/--parallel.
The original report suggested that this was due to the test dbs being created at the base dir instead of in the expected subdirs from the configured db names. That's not the case - although creating the test dbs at the base dir might be unexpected, it doesn't matter where the files are created, running multiple test suites at the same time will cause them all to attempt to create/use the db files from the same location.

I'm not convinced this is a case that Django should support - it would apply to other backends too, and would probably break multiple other things, as noted by previous comments.

The --keepdb/--parallel flags creating test dbs in an unexpected location is due to the inconsistent behaviour of keepdb with sqlite, which I've raised as a new ticket https://code.djangoproject.com/ticket/37371

I have a PR ready for the unable to open database file error.

[Becky Smith]

Note: the PR creates missing parent dirs on db creation, whether that's called by runserver or by tests, so the fix applies to both.
Other projects using sqlite have also encountered this and fixed it in a similar way, e.g. ​https://github.com/ploomber/ploomber/issues/281

[Johannes Maron]

Excellent stuff! BTW, patch is still missing a release note :)

[Jacob Walls]

Replying to Simon Charette:

Setting DATABASES['<alias>']['TEST']['NAME'] does not seem to solve this either.

I think we can accept on the basis that TEST.NAME should work.

I think this is what we have since solved separately in #36946.

---

Thanks for the reference to the peer project, Becky. Fixing the directory creation is in the same spirit as TEST.NAME should work, so I'm happy to repurpose this ticket for that and close it!

[GitHub <noreply@…>]

In 2f7f30c:

Fixed #36496 -- Ensured the parent path to a sqlite file exists.

Ensured a sqlite db path's parent directories are created if missing
before attempting to connect. OSErrors are reraised as InterfaceError.