Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#19856 closed Bug (fixed)

LiveServerTestCase + in-memory SQLite do not work together

Reported by: anonymous Owned by: nobody
Component: Testing framework Version: 1.4
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When running LiveServerTestCase with in-memory SQLite database (no NAME in settings) the liveserver and the test case share the connections. The static files serving done to by liveserver will cause concurrent requests while the test is running, and these concurrent requests will cause close_connection to be called (and also will cause session checking by cookies, and thus concurrent queries).

The browser is free to ask some static files even after the real request has finished. Such files might be images for example.

This causes two issues:

  • the newly introduced close_connection -> transaction.abort() call will cause constant spamming of transaction.rollback while the test case is running. This will cause random failures.
  • even if close_connection isn't called, there will be concurrent queries by two different threads, and at least some versions of sqlite do not like this at all (segfaults are possible).

Unfortunately this affects the just released 1.4.4. This only happens when running liveserver tests with in-memory SQLite DB. The workaround is to use file based SQLite when testing.

Change History (9)

comment:1 by Anssi Kääriäinen, 11 years ago

Triage Stage: UnreviewedAccepted

comment:2 by Julien Phalip, 11 years ago

I saw on IRC that you narrowed this down to the test not waiting for the next page to be loaded. Maybe something like this would be more robust: https://github.com/django/django/blob/master/tests/regressiontests/admin_inlines/tests.py#L550-L557

comment:3 by Anssi Kääriäinen, 11 years ago

Yeah, something along those lines is what I narrowed this down. The patch is this: https://github.com/akaariai/django/commit/aaa6eb91b2ef4dc564b04c2b6958965f5e7b870f

I tried to run the test admin_widgets.HorizontalVertical... test 10 times in a row with 10/10 success rate.

The transaction.abort() call does play a role in worsening this, as does static files serving. The static files serving launch request_finished() signal, and the concurrent page load will cause static files requests concurrently with the test. This makes the failure much louder as there will be transaction.abort() calls going on while test teardown is also going on. Or at least this is my current working theory, I have made so many false claims in this issue already that verifying my claims is a good idea...

request_finished should be removed from liveserver's static files serving, and likely also from the liveserver altogether if using a shared connection. Otherwise the transaction.abort() will ruin any transaction management going on in the test case itself. When not using shared connections, then the liveserver's connection should likely be closed, as otherwise it is possible that the liveserver's connection is still open when the tests finish.

All in all, transaction.abort() could cause some issues for users doing transaction management in their test case + using liveserver + using in-memory sqlite. This is hopefully an unlikely scenario. The test fix should be applied to 1.4-master, and likely use the try-except construct as used in admin_inlines tests.

comment:4 by Anssi Kääriäinen, 11 years ago

Has patch: set

Improved commit at https://github.com/akaariai/django/commit/88a253df90f26bd7a5ecbf9f45f1b50ca921fde1. I think this is ready for commit now.

comment:5 by Aymeric Augustin, 11 years ago

Patch needs improvement: set

I think wait_page_loaded should be a method of AdminSeleniumWebDriverTestCase.

comment:7 by Anssi Kääriäinen <akaariai@…>, 11 years ago

Resolution: fixed
Status: newclosed

In 50677b29af39ca670274fb45087415c883c78b04:

Made a couple of selenium tests wait for page loaded

The admin_widgets tests were issuing click() to the browser but
didn't wait for the effects of those clicks. This caused the resulting
request to be processed concurrently with the test case. When using
in-memory SQLite this caused weird failures.

Also added wait_page_loaded() to admin selenium tests for code
reuse.

Fixed #19856

comment:8 by Anssi Kääriäinen <akaariai@…>, 11 years ago

In 96790fc02214b4d36df0ecb630159e65b28a5174:

[1.5.x] Made a couple of selenium tests wait for page loaded

The admin_widgets tests were issuing click() to the browser but
didn't wait for the effects of those clicks. This caused the resulting
request to be processed concurrently with the test case. When using
in-memory SQLite this caused weird failures.

Also added wait_page_loaded() to admin selenium tests for code
reuse.

Fixed #19856, backpatch of 50677b29af39ca670274fb45087415c883c78b04

comment:9 by Anssi Kääriäinen <akaariai@…>, 11 years ago

In 3872bc51c966ac779f24772e24511423491ea49e:

[1.4.x] Made a couple of selenium tests wait for page loaded

The admin_widgets tests were issuing click() to the browser but
didn't wait for the effects of those clicks. This caused the resulting
request to be processed concurrently with the test case. When using
in-memory SQLite this caused weird failures.

Also added wait_page_loaded() to admin selenium tests for code
reuse.

Fixed #19856, cherry-pick of 50677b29af39ca670274fb45087415c883c78b04

Note: See TracTickets for help on using tickets.
Back to Top