Opened 3 years ago

Last modified 3 months ago

#29062 new Bug

"database table locked errors" when using sqlite in-memory database with LiveServerTestCase

Reported by: Juozas Masiulis Owned by: nobody
Component: Testing framework Version: 2.0
Severity: Normal Keywords: sqlite, testing, databases
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I am getting database table locked errors, when running tests which use sqlite in memory database.
This issue was introduced with this commit: https://github.com/django/django/commit/bece837829eafbc22f2598dadf82c9a8364b085a
I know this because reversing this commit resolves the issue for me.
If sqlite in memory testing is no longer supported, it should be mentioned in the documentation. If it is supported, there needs to be some parameter in the settings to use liveserver without multithreading. Or just make a new class liveservermultithreaded.

Change History (16)

comment:1 Changed 3 years ago by Tim Graham

Please provide a sample project that reproduces the problem.

comment:2 Changed 3 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed
Summary: Sqlite in memory database does not work for testing."database table locked errors" when using sqlite in-memory database with LiveServerTestCase

comment:3 Changed 3 years ago by Juozas Masiulis

https://github.com/Euphorbium/django-bug-demo here is the sample project. Sorry for the mess in the test, the issue is with multiprocessing. The test pass with my version of django, check requirements, and fails with the unmodified django.

comment:4 Changed 3 years ago by Juozas Masiulis

Resolution: needsinfo
Status: closednew

comment:5 Changed 2 years ago by Tim Graham

Resolution: worksforme
Status: newclosed

Is the error consistent? I ran your sample project ten times and the test usually passes. Once in a while, I see this failure:

======================================================================
FAIL: testStuff (bugdemo.tests.TestDatabaseLocking)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django-bug-demo/bugdemo/tests.py", line 75, in testStuff
    self.assertEqual(Question.objects.count(), 30)
AssertionError: 29 != 30

I think you'll have to explain why Django is at fault to move this report forward. Possibly the problem might be dependent on the Python version. I tested with Python 3.4.7, 3.5.5, and 3.6.4.

comment:6 Changed 2 years ago by Juozas Masiulis

Resolution: worksforme
Status: closednew

It is a race condition. It is consistent in my system, but it gets locked by node process, and I did not want to go outside of python to demonstrate the bug. It needs something faster to lock the database consistently.

comment:7 Changed 2 years ago by Tim Graham

Unfortunately your comment doesn't help with my understanding of the problem. Can explain why Django is at fault or propose a solution? If not, I'm not sure how to move this issue forward.

comment:8 Changed 2 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

comment:9 Changed 2 years ago by Juozas Masiulis

Resolution: needsinfo
Status: closednew

Have you tried running it with my version, which is in the requirements.txt? That NEVER fails. This is proof that it is a django problem, and it was introduced in https://github.com/django/django/commit/bece837829eafbc22f2598dadf82c9a8364b085a

comment:10 Changed 2 years ago by Tim Graham

Just because the behavior changed doesn't mean Django is at fault. You'll need to investigate and explain why Django is at fault. It might be that your testing code is at fault.

comment:11 Changed 2 years ago by Tim Graham

Triage Stage: UnreviewedSomeday/Maybe

comment:12 Changed 23 months ago by Andrew Badr

comment:13 Changed 23 months ago by Andrew Badr

Triage Stage: Someday/MaybeAccepted

Moving to 'Accepted' -- breaking test suites is a serious enough bug, and multiple people have hit it. Someday/Maybe is not appropriate here.

comment:14 Changed 16 months ago by Daniel Hahler

Just a drive-by comment since I've seen a similar issue and found this: check/ensure that transactions are enabled (i.e. with pytest-django transactional_db instead of db should be used).
Also check the code around https://github.com/django/django/commit/bece837829eafbc22f2598dadf82c9a8364b085a#diff-5d7d8ead1a907fe91ffc121f830f2a49L1287.

comment:15 in reply to:  12 Changed 13 months ago by Chris Spencer

Replying to Andrew Badr:

Yep, I'm hitting this too. See workaround at https://stackoverflow.com/questions/48353002/sqlite-database-table-is-locked-on-tests

This workaround does not work for me. I'm using the LiveServerTestCase to run a series of Selenium tests checking Ajax functionality, so the result of switching the LiveServerThread back to single-threaded WSGIServer only replaces the deadlock error with a timeout error. The Ajax calls timeout waiting for the main server thread to exit, which it never does.

comment:16 Changed 3 months ago by Matthijs Kooijman

What seems to be relevant here (might be obvious for others, but was not for me) is that Sqlite does not support a "wait until the lock is released" flow with the way Django implements transactions on Sqlite. This means that whenever you try to write to a database that already holds a write (EXCLUSIVE in Sqlite terms I think) lock, it will immediately fail rather than wait for the lock to be released. The underlying reason for this is described in https://code.djangoproject.com/ticket/29280

Note: See TracTickets for help on using tickets.
Back to Top