database threading issue with dev server and sqlite

[anonymous]

Steps to reproduce:

• Run dev server in the standard way "./manage.py runserver"
• Request a Django-served page in a browser
  • Likely important that the page has links to static media files served by the dev server so the browser initiates multiple simultaneous requests
  • There is middleware on the site that hits the session table in the db for every request, including the static media files
  • You may need to flush the browser cache when trying to repeatedly see the error

Expected behavior:

• All files served normally

Observed behavior:

• Non-deterministic, some requests cause an error
• Error response is the following: "A server error occurred. Please contact the administrator."
• The exception in the error log reads:

  DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 4374663168 and this is thread id 4385443840.
  

• Note that the Django site in question is only using the Django ORM, not reaching deeper into the django.db library, and it does not explicitly spawn any threads
• I'm including a full backtrace from the dev server below.

Environment:

• Mac OS X 10.6.8
• python27 @2.7.2_1 (MacPorts)
• py27-sqlite @2.6.3_0 (MacPorts)
• sqlite3 @3.7.11_0 (MacPorts)
• Django 1.4.0-final0 (pip)
• Firefox 11.0

Full backtrace:

DEBUG:django.db.backends:(0.003) SELECT "django_session"."session_key", "django_session"."session_data", "django_session"."expire_date" FROM "django_session" WHERE ("django_session"."session_key" = 02462878adc69e05312600990114b569  AND "django_session"."expire_date" > 2012-03-28 10:29:14.458986 ); args=('02462878adc69e05312600990114b569', u'2012-03-28 10:29:14.458986')
DEBUG:django.db.backends:(0.000) SELECT (1) AS "a" FROM "django_session" WHERE "django_session"."session_key" = 02462878adc69e05312600990114b569  LIMIT 1; args=('02462878adc69e05312600990114b569',)
DEBUG:django.db.backends:(0.000) UPDATE "django_session" SET "session_data" = NDY1YWVmMjI5ZTZiOGIxODUyMjVlMTAxOGQ2MjUzNTEzOWRmMWQ1NzqAAn1xAS4=
, "expire_date" = 2012-03-28 10:59:14.471961 WHERE "django_session"."session_key" = 02462878adc69e05312600990114b569 ; args=('NDY1YWVmMjI5ZTZiOGIxODUyMjVlMTAxOGQ2MjUzNTEzOWRmMWQ1NzqAAn1xAS4=\n', u'2012-03-28 10:59:14.471961', '02462878adc69e05312600990114b569')
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 67, in __call__
    return self.application(environ, start_response)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 243, in __call__
    signals.request_finished.send(sender=self.__class__)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django_digest/backend/db.py", line 54, in close_connection
    self.connection.close()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 319, in close
    self.validate_thread_sharing()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/__init__.py", line 136, in validate_thread_sharing
    % (self.alias, self._thread_ident, thread.get_ident()))
DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 4374663168 and this is thread id 4385443840.
[28/Mar/2012 10:29:14] "GET / HTTP/1.1" 500 59

Thanks very much, Django is great software!

[Anssi Kääriäinen]

I have looked into this a little, but I can't see where the connection is shared between the threads. Any ideas? It should be impossible to get the same DatabaseWrapper in different threads unless explicitly shared. Of course, it is possible there is a bug making it not so impossible, which would be more serious.

While I haven't reproduced this bug, I am still going to mark this accepted, as I have seen some strange threading errors, too (caused by using ipdb in my case, though).

[Anssi Kääriäinen]

I am going to revert the accepted triaging. I have been trying to reproduce this, and I can't. The method I have been using is a test view which does a database query and then sleeps a little. Then I hit that view continuously with high-concurrency AB benchmark. No errors.

In addition I have been reading the code, and I can't see a way the connection would get shared between threads.

So, more info is needed.

[Anssi Kääriäinen]

I can't reproduce this and I haven't seen other complaints about this. Closing this as worksforme.

[trey9000]

The bug turned out to be in the django_digest package that we were using.

See some discussion here:

​https://bitbucket.org/akoha/django-digest/issue/10/conflict-with-global-databasewrapper

A patched version of django-digest that works for me can be found here:

​https://github.com/trey0/django-digest

Sorry for reporting this in the wrong place...

[anonymous]

got the same exception when enable

from gevent import monkey; monkey.patch_all()

[anonymous]

Replying to anonymous:

got the same exception when enable

from gevent import monkey; monkey.patch_all()

Could be fixed by not patching threads:

 gevent.monkey.patch_all(socket=True, dns=True, time=True, select=True,thread=False, os=True, ssl=True, httplib=False, aggressive=True)

[idbill@…]

I am recently had this issue occur after I increased the logging level on a mod_wsgi instance.... I log quite a bit in debug mode, but rarely with that setting on anything other than in my development environment (which is usually running via runserver).