Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#19856 closed Bug (fixed)

LiveServerTestCase + in-memory SQLite do not work together

Reported by: anonymous Owned by: nobody
Component: Testing framework Version: 1.4
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When running LiveServerTestCase with in-memory SQLite database (no NAME in settings) the liveserver and the test case share the connections. The static files serving done to by liveserver will cause concurrent requests while the test is running, and these concurrent requests will cause close_connection to be called (and also will cause session checking by cookies, and thus concurrent queries).

The browser is free to ask some static files even after the real request has finished. Such files might be images for example.

This causes two issues:

  • the newly introduced close_connection -> transaction.abort() call will cause constant spamming of transaction.rollback while the test case is running. This will cause random failures.
  • even if close_connection isn't called, there will be concurrent queries by two different threads, and at least some versions of sqlite do not like this at all (segfaults are possible).

Unfortunately this affects the just released 1.4.4. This only happens when running liveserver tests with in-memory SQLite DB. The workaround is to use file based SQLite when testing.

Change History (9)

comment:1 Changed 2 years ago by akaariai

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 2 years ago by julien

I saw on IRC that you narrowed this down to the test not waiting for the next page to be loaded. Maybe something like this would be more robust: https://github.com/django/django/blob/master/tests/regressiontests/admin_inlines/tests.py#L550-L557

comment:3 Changed 2 years ago by akaariai

Yeah, something along those lines is what I narrowed this down. The patch is this: https://github.com/akaariai/django/commit/aaa6eb91b2ef4dc564b04c2b6958965f5e7b870f

I tried to run the test admin_widgets.HorizontalVertical... test 10 times in a row with 10/10 success rate.

The transaction.abort() call does play a role in worsening this, as does static files serving. The static files serving launch request_finished() signal, and the concurrent page load will cause static files requests concurrently with the test. This makes the failure much louder as there will be transaction.abort() calls going on while test teardown is also going on. Or at least this is my current working theory, I have made so many false claims in this issue already that verifying my claims is a good idea...

request_finished should be removed from liveserver's static files serving, and likely also from the liveserver altogether if using a shared connection. Otherwise the transaction.abort() will ruin any transaction management going on in the test case itself. When not using shared connections, then the liveserver's connection should likely be closed, as otherwise it is possible that the liveserver's connection is still open when the tests finish.

All in all, transaction.abort() could cause some issues for users doing transaction management in their test case + using liveserver + using in-memory sqlite. This is hopefully an unlikely scenario. The test fix should be applied to 1.4-master, and likely use the try-except construct as used in admin_inlines tests.

comment:4 Changed 2 years ago by akaariai

  • Has patch set

Improved commit at https://github.com/akaariai/django/commit/88a253df90f26bd7a5ecbf9f45f1b50ca921fde1. I think this is ready for commit now.

comment:5 Changed 2 years ago by aaugustin

  • Patch needs improvement set

I think wait_page_loaded should be a method of AdminSeleniumWebDriverTestCase.

comment:7 Changed 2 years ago by Anssi Kääriäinen <akaariai@…>

  • Resolution set to fixed
  • Status changed from new to closed

In 50677b29af39ca670274fb45087415c883c78b04:

Made a couple of selenium tests wait for page loaded

The admin_widgets tests were issuing click() to the browser but
didn't wait for the effects of those clicks. This caused the resulting
request to be processed concurrently with the test case. When using
in-memory SQLite this caused weird failures.

Also added wait_page_loaded() to admin selenium tests for code
reuse.

Fixed #19856

comment:8 Changed 2 years ago by Anssi Kääriäinen <akaariai@…>

In 96790fc02214b4d36df0ecb630159e65b28a5174:

[1.5.x] Made a couple of selenium tests wait for page loaded

The admin_widgets tests were issuing click() to the browser but
didn't wait for the effects of those clicks. This caused the resulting
request to be processed concurrently with the test case. When using
in-memory SQLite this caused weird failures.

Also added wait_page_loaded() to admin selenium tests for code
reuse.

Fixed #19856, backpatch of 50677b29af39ca670274fb45087415c883c78b04

comment:9 Changed 2 years ago by Anssi Kääriäinen <akaariai@…>

In 3872bc51c966ac779f24772e24511423491ea49e:

[1.4.x] Made a couple of selenium tests wait for page loaded

The admin_widgets tests were issuing click() to the browser but
didn't wait for the effects of those clicks. This caused the resulting
request to be processed concurrently with the test case. When using
in-memory SQLite this caused weird failures.

Also added wait_page_loaded() to admin selenium tests for code
reuse.

Fixed #19856, cherry-pick of 50677b29af39ca670274fb45087415c883c78b04

Note: See TracTickets for help on using tickets.
Back to Top