#34825 closed Bug (fixed)
SQLite database files are not destroyed after tests
Reported by: | Jacob Walls | Owned by: | David Sanders |
---|---|---|---|
Component: | Testing framework | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Ready for checkin | |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
After running the Django unit tests, I'm left with 8 .sqlite3 files like so.
Bisected to 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.
tests % python3.11 runtests.py admin_inlines Testing against Django installed in '/Users/jwalls/django/django' with up to 8 processes Found 82 test(s). Creating test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... System check identified no issues (0 silenced). .ssssssssssss..................................................................... ---------------------------------------------------------------------- Ran 82 tests in 1.419s OK (skipped=12) Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... tests % git status On branch main Your branch is ahead of 'origin/main' by 86 commits. (use "git push" to publish your local commits) Untracked files: (use "git add <file>..." to include in what will be committed) other_1.sqlite3 other_2.sqlite3 other_3.sqlite3 other_4.sqlite3 other_5.sqlite3 other_6.sqlite3 other_7.sqlite3 other_8.sqlite3 nothing added to commit but untracked files present (use "git add" to track)
Interestingly, I set a breakpoint at _destroy_test_db()
and found this:
(Pdb) test_database_name 'file:memorydb_default?mode=memory&cache=shared'
Change History (19)
comment:1 by , 14 months ago
comment:2 by , 14 months ago
I'm new to the test framework but here's what I found when doing some investigating:
Looks like the issue is with the new setup_worker_connection()
method for the sqlite3 backend in that commit you bisected.
It attempts to copy from other_<worker-id>.sqlite3
into memory … but it doesn't exist because the test runner doesn't create databases for the "other" alias. The default behaviour of sqlite3.connect()
is to create a database because the default mode is mode=rwc
, c
being for create.
We can apparently change the mode to mode=rw
in which case an exception is raised. Or we can adjust the logic to use the same settings as "fork" if the file doesn't exist. 🤔
(I just quickly tested this theory by appending mode=rw and it no longer creates the other_n.sqlite3 files)
comment:3 by , 14 months ago
Has patch: | set |
---|---|
Triage Stage: | Unreviewed → Accepted |
PR demonstrating my comment: ​https://github.com/django/django/pull/17242/files
comment:4 by , 14 months ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:5 by , 14 months ago
I'm not being able to reproduce, I'm using python3.11 with latest main
on a Manjaro system:
$ python3.11 runtests.py admin_inlines Testing against Django installed in '/home/nessita/fellowship/django/django' with up to 4 processes Found 82 test(s). Creating test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... System check identified no issues (0 silenced). ..............ssssssssssss........................................................ ---------------------------------------------------------------------- Ran 82 tests in 1.928s OK (skipped=12) Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'... $ git st On branch main Your branch is up to date with 'origin/main'. nothing to commit, working tree clean $ ls *sqlite* test_sqlite.py
comment:6 by , 14 months ago
I'm not being able to reproduce, I'm using python3.11 with latest main on a Manjaro system:
It's an issue with "spawn" method, so it's reproducible on Windows and MacOS.
comment:7 by , 14 months ago
I haven't tried this but you might also be able to set the start method in runtests.py and change to spawn: ​https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method
comment:8 by , 14 months ago
Thank you everyone, I was able to reproduce with this diff:
-
tests/runtests.py
diff --git a/tests/runtests.py b/tests/runtests.py index 2eb7490170..ad63e6a286 100755
a b if __name__ == "__main__": 744 744 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_sqlite") 745 745 options.settings = os.environ["DJANGO_SETTINGS_MODULE"] 746 746 747 multiprocessing.set_start_method("spawn") 747 748 if options.selenium: 748 749 if multiprocessing.get_start_method() == "spawn" and options.parallel != 1: 749 750 parser.error(
comment:9 by , 14 months ago
Triage Stage: | Accepted → Ready for checkin |
---|
comment:11 by , 14 months ago
Hi faizan2700, unfortunately it's already assigned & fixed. Your best bet is to search through tickets that are unassigned if you'd like to contribute.
comment:12 by , 14 months ago
Triage Stage: | Ready for checkin → Accepted |
---|
Setting as Accepted until the conversation in the PR are resolved.
comment:13 by , 14 months ago
Patch needs improvement: | set |
---|
comment:14 by , 14 months ago
Patch needs improvement: | unset |
---|---|
Triage Stage: | Accepted → Ready for checkin |
What's interesting is if you do
--parallel=1
then they aren't left around 🤔