#24080 closed Bug (invalid)
Sqlite segmentation fault regression when running tests (since 1.7.2)
Reported by: | Steve Jalim | Owned by: | nobody |
---|---|---|---|
Component: | Testing framework | Version: | dev |
Severity: | Normal | Keywords: | sqlite nose |
Cc: | Shai Berger, tomas.ehrlich@…, chipx86@…, Matt W, Joseph Gordon, Caio Ariede, scenting@…, mvbeacom@…, dandamian@…, Miguel Sánchez, bsd@…, jbzdak@…, davidhalter88+django@…, kontakt@… | Triage Stage: | Accepted |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
A project has a reasonably large number of tests (~1600). As part of our dev workflow, we run tests against sqlite first, then against postgres before checkin.
On 1.7.1 all tests pass on sqlite, bar some deliberate DB-specific exclusions for tests that need specific postgres features. They also all pass on postgres.
On 1.7.2, all tests still pass on postgres (though the server needed a restart) but the sqlite test run fails at varying points with "segmentation fault" (and an exit code of 139, accordingly).
The segmentation fault occurs regardless of whether an in-memory or file-based database is used. When the file-based run crashes, the DB is around 495k in size.
There is very little output, even at max verbosity, because this is coming from something lower down than Django itself. However, because it only started happening with 1.7.2 and goes away when we revert to 1.7.1, I am assuming a change somewhere caused the regression.
Setup spec:
- Headless Virtualbox + Vagrant with 1GB allocated RAM
- Ubuntu 14.04 LTS, fully up to date
- libsqlite3-0 version: 3.8.2-1ubuntu2 amd64
- Python: 2.7.5 (up-to-date version for 14.04 - unclear which parts, if any of 2.7.8 have been backported)
Any steers here would be much appreciated. apollo13 on #django pointed out that I could git bisect the commits between 1.7.1 and 1.7.2, which I am open to doing if I can make the time.
Cheers
Steve
Change History (60)
comment:1 by , 10 years ago
Description: | modified (diff) |
---|
comment:2 by , 10 years ago
comment:3 by , 10 years ago
Cc: | added |
---|
gdb
can be used to debug Python programs at both Python and C levels; this may help you.
See https://docs.python.org/devguide/gdb.html and https://wiki.python.org/moin/DebuggingWithGdb
Another info which may be even more useful is to pinpoint a test which causes the segfault. If you can share code (I assume, smaller than your entire project) which causes the failure it will be very helpful.
If possible, try running your code on other versions of Python -- non-Ubuntu, or even Python3. You may be able to get a consistent segfault that way.
comment:4 by , 10 years ago
Thanks shaib - I'll see what gdb can turn up.
Yes, I'd already considered providing a sample project, but that's not practical as the segfault appears to happen in different places/times in the suite run and a small run does not trigger it. I shall keep digging.
comment:5 by , 10 years ago
Resolution: | → needsinfo |
---|---|
Status: | new → closed |
follow-up: 7 comment:6 by , 10 years ago
Just as an update for the future: I 'solved' this by throwing more RAM at it. Haven't seen it happen since.
comment:7 by , 10 years ago
Cc: | added |
---|---|
Component: | Uncategorized → Testing framework |
Resolution: | needsinfo |
Status: | closed → new |
Triage Stage: | Unreviewed → Accepted |
This regression is caused by #20392. In the ticket there's a mention about sqlite:
I also ran into some weird threading issues with sqlite that was causing a segmentation fault.
For me it happens in OS X 10.10.2, Python 2.7.6 and latest master branch.
comment:8 by , 10 years ago
Type: | Uncategorized → Bug |
---|---|
Version: | 1.7 → master |
comment:9 by , 10 years ago
The original report is against Django 1.7.2 -- the change for the linked ticket is only present on 1.8+.
comment:11 by , 10 years ago
It's probably related only to Python 2.7.6. In Python 3.4.1 the test case runs fine.
Unfortunately I haven't found the cause of segmentation fault. Here is the traceback (begins with the last test case):
SAVEPOINT "s140735267844864_x61" test_use_explicit_o2o_to_parent_from_abstract_model (model_inheritance_regress.tests.ModelInheritanceTest) ... ok ROLLBACK TO SAVEPOINT "s140735267844864_x61" Fatal Python error: Segmentation fault Current thread 0x00007fff7ba5b300 (most recent call first): File "/Users/tricoder/Projects/django/django/db/backends/sqlite3/base.py", line 317 in execute File "/Users/tricoder/Projects/django/django/db/backends/utils.py", line 63 in execute File "/Users/tricoder/Projects/django/django/db/backends/base/base.py", line 215 in _savepoint_rollback File "/Users/tricoder/Projects/django/django/db/backends/base/base.py", line 255 in savepoint_rollback File "/Users/tricoder/Projects/django/django/db/transaction.py", line 252 in __exit__ File "/Users/tricoder/Projects/django/django/test/testcases.py", line 932 in _rollback_atomics File "/Users/tricoder/Projects/django/django/test/testcases.py", line 988 in _fixture_teardown File "/Users/tricoder/Projects/django/django/test/testcases.py", line 844 in _post_teardown File "/Users/tricoder/Projects/django/django/test/testcases.py", line 184 in __call__ File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py", line 108 in run File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py", line 70 in __call__ File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py", line 151 in run File "/Users/tricoder/Projects/django/django/test/runner.py", line 178 in run_suite File "/Users/tricoder/Projects/django/django/test/runner.py", line 211 in run_tests File "./tests/runtests.py", line 236 in django_tests File "./tests/runtests.py", line 414 in <module> [1] 38431 segmentation fault ./tests/runtests.py -v3 model_inheritance_regress
Some clues:
- Good library for catching tracebacks on segfault is
faulthandler
http://faulthandler.readthedocs.org/ - Some failing test suites are
model_inheritance_regress
(with traceback above) andadmin_views
- It always crashes on "ROLLBACK TO SAVEPOINT", even in different tests (admin_views for example).
- It doesn't depend on how many savepoints is created and released (
admin_views
tests create about 350 savepoints, whilemodel_inheritance_regress
creates about 60 savepoints)
comment:12 by , 10 years ago
I can confirm that updating from default 2.7.6 on OS X 10.10 to 2.7.9 gets rid of the error.
I did notice that Homebrew also installed an updated version of sqlite3; so I can't say for 100% sure if it was just the Python update or the updated sqlite3 that Homebrew binds to Python 2.7.9.
comment:13 by , 10 years ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
As Django only officially supports the latest micro release, I think we can close this.
comment:14 by , 10 years ago
Resolution: | invalid |
---|---|
Status: | closed → new |
i can reproduce on 2.7.9
$ python --version Python 2.7.9 $ python ./tests/runtests.py -v2 admin_views.tests.AdminViewBasicTest # [snip] ... test_sort_indicators_admin_order (admin_views.tests.AdminViewBasicTest) ... ok Segmentation fault: 11
with faulthandler
:
$ python ./tests/runtests.py -v2 admin_views.tests.AdminViewBasicTest [snip] Current thread 0x00007fff791a1300 (most recent call first): File "/Users/david/dev/django/django/db/backends/sqlite3/base.py", line 316 in execute File "/Users/david/dev/django/django/db/backends/utils.py", line 62 in execute File "/Users/david/dev/django/django/db/backends/base/base.py", line 214 in _savepoint_rollback File "/Users/david/dev/django/django/db/backends/base/base.py", line 254 in savepoint_rollback File "/Users/david/dev/django/django/db/transaction.py", line 252 in __exit__ File "/Users/david/dev/django/django/test/testcases.py", line 950 in _rollback_atomics File "/Users/david/dev/django/django/test/testcases.py", line 1003 in _fixture_teardown File "/Users/david/dev/django/django/test/testcases.py", line 862 in _post_teardown File "/Users/david/dev/django/django/test/testcases.py", line 190 in __call__ File "/Users/david/.pythonz/pythons/CPython-2.7.9/lib/python2.7/unittest/suite.py", line 108 in run File "/Users/david/.pythonz/pythons/CPython-2.7.9/lib/python2.7/unittest/suite.py", line 70 in __call__ File "/Users/david/.pythonz/pythons/CPython-2.7.9/lib/python2.7/unittest/runner.py", line 151 in run File "/Users/david/dev/django/django/test/runner.py", line 178 in run_suite File "/Users/david/dev/django/django/test/runner.py", line 211 in run_tests File "./tests/runtests.py", line 256 in django_tests File "./tests/runtests.py", line 434 in <module>
comment:15 by , 10 years ago
a few more data points
can reproduce with this subset of tests (but removing any more make the run pass)
admin_views.tests.AdminViewBasicTest.test_allows_attributeerror_to_bubble_up admin_views.tests.AdminViewBasicTest.test_change_list_null_boolean_display admin_views.tests.AdminViewBasicTest.test_change_list_sorting_model_meta admin_views.tests.AdminViewBasicTest.test_change_view_with_show_delete_extra_context admin_views.tests.AdminViewBasicTest.test_changelist_with_no_change_url admin_views.tests.AdminViewBasicTest.test_disallowed_filtering admin_views.tests.AdminViewBasicTest.test_multiple_sort_same_field admin_views.tests.AdminViewBasicTest.test_popup_dismiss_related admin_views.tests.AdminViewBasicTest.test_proxy_model_content_type_is_used_for_log_entries admin_views.tests.AdminViewBasicTest.test_sort_indicators_admin_order
separately, setting TEST
in DATABASES
to use a sqlite db on disk instead of in ram seems to fix the issue
comment:16 by , 10 years ago
FWIW when I encountered this, I found that:
1) It took a non-small number of tests as part of the run to trigger it (I didn't get as far as a firm number)
2) my project hit the segfault regardless of in-memory or disk-based database
Hope that helps,
S
comment:17 by , 10 years ago
I ran into this as well and did a bunch of testing:
- I'm running on OS X Yosemite 10.10.2. I can consistently replicate by running admin_views.tests.AdminViewBasicTest on the latest master branch of Django.
- I was able to replicate with Python 2.7.9 and 3.4.1, both built with pyenv.
- I was able to replicate with the system python 2.7.6.
- I was unable to replicate with an older version of Python installed via homebrew, but that was because it didn't have sqllite built into Python.
- I was unable to replicate with a fresh install of Python 2.7.9 from homebrew. When I upgraded it also upgraded my homebrew-installed version of sqlite (or whatever it's called) to the latest version as well.
I suspect the issue is related to the sqlite version. I can't figure out which sqlite version pyenv is using when it builds python, but it might be useful for anyone affected to try re-building python with a recent version of sqlite to see if they still get the issue.
comment:18 by , 10 years ago
If this can be of any help, I ran these tests on Mac OSX and Ubuntu Linux,
On Mac OSX with Python 3.4.3 inside a virtualenv,
$ python
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Ran 9273 tests in 260.431s
OK (skipped=603, expected failures=6)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...
On Ubuntu Linux 14.04,
$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:38)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Ran 9275 tests in 231.808s
OK (skipped=548, expected failures=7)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...
However, on Mac OSX using the built-in python 2.7.6,
............................................................................................<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.....................................................................................................................................................................................................................................[1] 52476 segmentation fault
Results :
Tests are successful on Mac OSX python 3.4.3 and Ubuntu Linux (14.04) 2.7.6.
Test fails on Mac OSX 2.7.6.
comment:19 by , 10 years ago
From what I learned from the sprint it looks like homebrew builds Python with the currently installed version of SQLite hence the failures on different versions here.
What should be really useful to isolate the bug would be to try rebuilding Python with different versions of SQLite, create a virtualenv with this newly installed python interpreter and run something along those lines:
/path/to/intepreter/python -c" import sqlite3 connection = sqlite3.connect(':memory:') cursor = connection.cursor() cursor.execute('SELECT sqlite_version()') print(cursor.fetchone()) "
To try to figure out which version of SQLite provided by Homebrew we should advise not to build Python with.
comment:20 by , 9 years ago
On Mac OS 10.9.5, using MacPorts, python 3.3.6, I get a segfault with SQLite 3.8.8.2, but not with SQLite 3.8.7.4. I had a segfault (during messages in my case, but it doesn't appear to be "precise test related"), switched with "port activate sqlite3 @3.8.7.4_0", and no longer had a segfault.
comment:21 by , 9 years ago
Did some git bisecting, and found the commit which introduced this issue: https://github.com/django/django/commit/da9fe5c717c179a9e881a40efd3bfe36a21bf4a6
Investigating a fix now, but not 100% sure on what's going on. Commenting out the calls to these two methods prevents the segfault, but triggers a whole ton of other issues: https://github.com/gchp/django/blob/master/django/test/testcases.py#L967-L981
comment:22 by , 9 years ago
I've found that commit too https://code.djangoproject.com/ticket/24080#comment:7, but don't know how to fix it. It seems it's related to python version on OS X
comment:23 by , 9 years ago
When building python with pyenv, it uses the system version of sqlite3 (3.8.5 in Yosemite), and the segmentation fault occurs.
To avoid this, install the latest version of sqlite from homebrew (3.8.10.2 at the time of this writing):
$ brew install sqlite ==> Downloading https://homebrew.bintray.com/bottles/sqlite-3.8.10.2.yosemite.bottle.tar.gz ######################################################################## 100.0% ==> Pouring sqlite-3.8.10.2.yosemite.bottle.tar.gz ==> Caveats This formula is keg-only, which means it was not symlinked into /usr/local. Mac OS X already provides this software and installing another version in parallel can cause all kinds of trouble. OS X provides an older sqlite3. Generally there are no consequences of this for you. If you build your own software and it requires this formula, you'll need to add to your build variables: LDFLAGS: -L/usr/local/opt/sqlite/lib CPPFLAGS: -I/usr/local/opt/sqlite/include ==> Summary 🍺 /usr/local/Cellar/sqlite/3.8.10.2: 9 files, 2.8M
And before installing a new python version with pyenv, execute this (the paths are taken from the output above):
$ export LDFLAGS="-L/usr/local/opt/sqlite/lib" $ export CPPFLAGS="-I/usr/local/opt/sqlite/include" $ pyenv install 2.7.10
The compiled python interpreter will use the newer sqlite version from homebrew, and no more segfaults!
comment:24 by , 9 years ago
Cc: | added |
---|
comment:25 by , 9 years ago
So this seems to be hitting us too on our Ubuntu 14.04 build hosts. If we run the tests on our Ubuntu 12.04 hosts, we don't seem to have the problem. I've tried upgrading Sqlite to the latest available release (3.8.2) with no luck. I also tried to force the test DB runner to use disk-backed sqlite databases, but ran into issue https://github.com/django-nose/django-nose/issues/97.
Any news on this issue?
comment:26 by , 9 years ago
comment:27 by , 9 years ago
Cc: | added |
---|
comment:28 by , 9 years ago
Cc: | added |
---|
comment:29 by , 9 years ago
Cc: | added |
---|
comment:30 by , 9 years ago
We're running into the same issue with our project tests suite. 100% reproducible. Django 1.8.3, Debian Jessie + Python 2.7.10, using Docker.
I tried:
- using latest pysqlite2 version 2.6.3 instead of sqlite3 version 2.6.0 from Python standard library;
- doubling RAM size for the virtual machine which is running tests;
- upgrading sqlite3 version from 3.8.7.1 to 3.8.11.1.
None of above worked for me. Finally, I tried running it from gdb and it shows Segmentation fault in sqlite3.c, but I have no idea what can I do with it:
(gdb) run manage.py test api.tests.test_member --settings api.settings Starting program: /usr/local/bin/python manage.py test api.tests.test_member --settings api.settings [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Traceback (most recent call last): File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20-gdb.py", line 59, in <module> from libstdcxx.v6.printers import register_libstdcxx_printers File "/usr/lib/x86_64-linux-gnu/../../share/gcc-4.9/python/libstdcxx/v6/printers.py", line 19, in <module> import itertools ImportError: No module named itertools Creating test database for alias 'default'... /usr/local/lib/python2.7/site-packages/redis_cache/utils.py:4: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9. from django.utils.importlib import import_module .. Program received signal SIGSEGV, Segmentation fault. memjrnlRead (pJfd=0x158a058, zBuf=<optimized out>, iAmt=4, iOfst=98552) at sqlite3.c:82326 82326 memcpy(zOut, &pChunk->zChunk[iChunkOffset], nCopy);
comment:31 by , 9 years ago
Cc: | added |
---|
comment:32 by , 9 years ago
I encountered this error on an Ubuntu 12.04 server running Python 2.7.3 with Django 1.8.3 during jenkins testing.
I attempted to resolve the issue by updating Python to 2.7.10. Due to my server running 12.04, I was unable to upgrade to the latest SQLite3 (sqlite3.sqlite_version: '3.8.11.1') via APT due to the package restrictions.
Ultimately, I resolved the issue by upgrading manually via SQLite 3.8.11.1 source (https://www.sqlite.org/2015/sqlite-autoconf-3081101.tar.gz) and apt-get install --reinstall'ing python 2.7.10!
You can install SQLite 3.8.11.1 by building from the previously mentioned source .tar.gz:
$ wget https://www.sqlite.org/2015/sqlite-autoconf-3081101.tar.gz $ tar xvfz sqlite-autoconf-3081101.tar.gz $ cd sqlite-autoconf-3081101 $ ./configure --prefix=/usr/local $ make $ make install $ sudo apt-get install --reinstall python2.7 python2.7-minimal python2.7-dev python2.7-dbg
comment:33 by , 9 years ago
Cc: | added |
---|
comment:34 by , 9 years ago
Cc: | added |
---|
comment:35 by , 9 years ago
Our software organization is also having the same issues outlined in this bug report, and I wanted to expand on my own findings.
The issue: Segfault in running unit tests which haven't changed in months to years when upgrading from Django 1.6.x to Django 1.8.x
Running faulthandler prior to py.test reveals the same kind of stack trace everyone is posting which is essentially that in the new _rollback_atomics code that happens it causes the seg-fault. I break-pointed some of the code and generally the error happens for us when we catch an IntegrityError and handle it outside of an atomic() block. The thing is running the same test in isolation allows the test it segfaulted on to pass, but running in the group it causes the failure.
I have experienced the failure originally on:
Ubuntu 14.04 LTS with Python 2.7 and sqlite3 3.8.7.1.
Upgraded sqlite3 3.8.7.1 to 3.8.11.1 as outlined by stebunovd, but issue persisted.
Installed Ubuntu 15.04 w/ Python 2.7 and sqlite3 3.8.7.1 (default) to a VM and the issue persisted.
Upgraded Ubuntu 15.04 sqlite3 3.8.7.1 to 3.8.11.1 as outlined by stebunovd, but issue persisted.
In both cases I ensured that the old version of sqlite3 was removed and python was rebuilt after the CLI returned the updated version info for sqlite3.
One interesting point to note is that upgrading sqlite3 did postpone the segmentation fault until later in the test suite, but it too happened in the same manner suggesting that the new version doesn't actually fix the problem here.
comment:36 by , 9 years ago
Just wanted to include also my stack trace from faulthandler running on the latest everything 15.04:
Current thread 0x00007f6d07308700 (most recent call first):
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 316 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 62 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 214 in _savepoint_rollback
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254 in savepoint_rollback
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/transaction.py", line 252 in exit
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 941 in _rollback_atomics
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 998 in _fixture_teardown
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 853 in _post_teardown
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 189 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/unittest.py", line 137 in runtest
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 90 in pytest_runtest_call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 123 in init
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 107 in wrapped_call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 393 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 137 in <lambda>
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 149 in init
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 137 in call_runtest_hook
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 119 in call_and_report
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 75 in runtestprotocol
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 65 in pytest_runtest_protocol
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 123 in init
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 107 in wrapped_call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 393 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 142 in pytest_runtestloop
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 122 in _main
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 84 in wrap_session
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 116 in pytest_cmdline_main
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.pycharm_helpers/pycharm/pytestrunner.py", line 35 in main
File "/home/matt/.pycharm_helpers/pycharm/pytestrunner.py", line 60 in <module>
bash: line 1: 4383 Segmentation fault (core dumped) env
comment:37 by , 9 years ago
Cc: | added |
---|
comment:38 by , 9 years ago
I am also encountering this error. In my case, there are three specific tests, which if I skip, avoid the problem. It is entirely reproducible, the same test will segfault every time if I don't skip it.
I'm using Python 2.7.3 on ubuntu 12.04, with sqlite.version == "2.6.0" and sqlite.sqlite_version == "3.7.9"
comment:39 by , 9 years ago
Hi Ned,
Are you able to show us an entire test that is causing problems? It might help if we had examples to look at.
comment:40 by , 9 years ago
This is affecting us also, randomly dying during our automated tests. It is entirely repeatable, but unpredictable where it will break. It happens both locally on our Macs (sqlite v3.7.13) and on Linux. Upgrading to sqlite 3.8 just makes it worse. It happens between tests, where Django tries to rollback the test transaction in _rollback_atomics. The only reliable solution is to stop using Django's TestCase class and instead use the one from unittest, manually deleting in tearDown() any models that the test creates.
comment:41 by , 9 years ago
@jarshwah sorry, didn't see your earlier comment. The test suite is on Open edX, which is one of those unwieldy projects that are hard to hand over as reproducible test cases. I can walk you through how to get it set up in a Vagrant instance if you want, it isn't *that* hard.
comment:42 by , 9 years ago
Just to confirm the @jcugat solution: I was having runtests.py
crashing with cryptic errors on OS X 10.11.1. After installing a new sqlite version via brew, errors are gone.
comment:43 by , 9 years ago
Cc: | added |
---|
comment:44 by , 9 years ago
I ran into this after an update to django1.8.6. I was able to trace it down to a test that wasn't crashing but was missing a super(MyTest, cls).tearDownClass()
. The test that actually crashed used override_settings
and adding either the super tearDownClass
or removing the override_settings
resolved the issue for me.
Unfortunately, coming up with an example is proving hard because I'm only seeing the segfault much later (I don't see the error when running the test with a missing tearDownClass
and then the test with override_settings
even though I see the error when I run the full suite).
class MyTest(TestCase): @classmethod def tearDownClass(cls): pass ... @override_settings(...) class MyOtherTest(TestCase): pass
I'll try to distill this to a full example.
comment:45 by , 9 years ago
#25860 seems to be a duplicate and also points to a case where django.test.TestCase
fails to be called (if skipTest
is raised in setUpClass()
. The original report is about a regression in 1.7.2 but the subsequent reports seem to be for 1.8 and related to setUpTestData()
so it seems we should retitle this issue, create a new one, or use #25860 (closing this one as needsinfo since we still don't have anything to reproduce a problem on 1.7.2). It's not immediately to me if/how we'd fix the problem since it's somewhat related to user code, but maybe we can at least do better to detect the situation or document it.
comment:46 by , 9 years ago
Keywords: | nose added |
---|
Just a small note as I also ran into this issue (an exception in backends/sqlite3/base.py when running tests with nose) It's difficult to exactly pinpoint versions of python/sqlite/django as it's behaving differently on different platforms.
However, I managed to fix the segmentation fault by changing the TestCase as following:
from django.test import TransactionTestCase as TestCase class MyTestCase(TestCase): """My segfaulting testcase""" cleans_up_after_itself = True
So I suspect something with cleaning up after a testcase is finished. (See also http://django-nose.readthedocs.org/en/latest/usage.html#escaping-the-grime)
comment:47 by , 9 years ago
Cc: | added |
---|
comment:48 by , 9 years ago
Were able to fix this issue by calling 'flush' command in tearDown. This behaviour was taken from TransactionTestCase._fixture_teardown()
call_command('flush', verbosity=0, interactive=False, reset_sequences=False, allow_cascade=False, inhibit_post_migrate=False)
comment:49 by , 9 years ago
As far as I can tell, all the fixes in this thread are workarounds that - at least temporarily - keep the segmentation fault from manifesting itself while not addressing the root cause.
Now, I don't have a solution myself. But I just want to point out that creating a dummy boolean value to trigger a subtle change in the way memory is allocated does not guarantee that the bug won't be triggered again and that it does nothing to address the underlying issue.
comment:50 by , 9 years ago
Since Django doesn't contain any C code, a segfault must be a bug in Python or in another third-party software. The bug can be caused by the software itself or by modifications performed by distributors.
The purpose of this ticket is to identify the issue more precisely, so it can be reported upstream, and to identify workarounds in the meantime. It's highly unlikely to result in changes to Django.
comment:51 by , 9 years ago
Since the switch to SQLite 3.12 I can reproduce a similar segfault (same stacktrace output by faulthandler) with Django 1.9.5 and Python 2.7.11 or 3.5.1.
$ cd tests $ PYTHONPATH=.. python3 -X faulthandler ./runtests.py --verbosity=2 --parallel=1 admin_views.tests.AdminViewBasicTest [...] test_popup_dismiss_related (admin_views.tests.AdminViewBasicTest) ... ok Fatal Python error: Segmentation fault Current thread 0x00007fab40575700 (most recent call first): File "/home/rhertzog/deb/pkg/python-django/django/db/backends/sqlite3/base.py", line 321 in execute File "/home/rhertzog/deb/pkg/python-django/django/db/backends/utils.py", line 62 in execute File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", line 288 in _savepoint_rollback File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", line 328 in savepoint_rollback File "/home/rhertzog/deb/pkg/python-django/django/db/transaction.py", line 243 in __exit__ File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 1016 in _rollback_atomics File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 1073 in _fixture_teardown File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 919 in _post_teardown File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 217 in __call__ File "/usr/lib/python3.5/unittest/suite.py", line 122 in run File "/usr/lib/python3.5/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.5/unittest/runner.py", line 176 in run File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 494 in run_suite File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 533 in run_tests File "./runtests.py", line 275 in django_tests File "./runtests.py", line 458 in <module>
Maybe SQLite 3.12 makes it more likely to trigger to the problem or something like that...
comment:54 by , 8 years ago
Colleague of mine discovered a workaround - try renaming test methods to change their execution order. For example, rename the method on which Segmentation fault occurs to be the first in its test class (in alphabetical order). Sounds weird, but it worked both for him and for me.
comment:55 by , 8 years ago
My experience with this issue is mostly related to ManyToMany fields.
I even traced all the SQL queries, and apparently sqlite does not handle the cleanup phase well and segfaults when truncating the data when destroying database. I can try reproduce this and post the log, if this can be useful.
I managed to solve that by clearing many to many fields in tearDown
:
def tearDown(self): self.user.sites.clear() for post in Post.objects.all(): post.sites.clear() post.tags.clear() super(BaseTest, self).tearDown()
comment:56 by , 8 years ago
Cc: | added |
---|
comment:57 by , 8 years ago
Cc: | added |
---|
comment:58 by , 8 years ago
I also see this issue (Django 1.9.9, OSX, Python 2.7.11) and confirm that updating from OSX built-in sqlite3 3.8.5 to homebrew sqlite3 3.14.0 fixes this issue, as well as fixes the sqlite repro case detailed in https://www.sqlite.org/src/info/7f7f8026eda38
My shell notes: https://gist.github.com/jasonm/5aa9406c88e29e3d550c1b42f9435cf2
comment:59 by , 8 years ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
I think we can close this issue as nothing indicates a problem in Django itself and SQLite has made a fix that seems to help.
comment:60 by , 8 years ago
On a linux distribution(ubunutu) this gist can help installing latest(3.14) sqlite.https://gist.github.com/ziafazal/cf2b10b38b39e1795d5d8eaa993760ce
I think there is little chance we can help without additional information, especially the git bisect results.