id summary reporter owner description type status component version severity resolution keywords cc stage has_patch needs_docs needs_tests needs_better_patch easy ui_ux 18251 multithreading deadlock in django.models.loading.get_apps harm nobody " On a production site we have encountered deadlocks, which after fierce investigation pointed to the django internals. We use a deployment: apache/mod_wsgi apache config: {{{ WSGIDaemonProcess processes=1 threads=10 umask=0002 display-name=%{GROUP} stack-size=524288 }}} == details of deadlock == During the bootstrap of django, so the first requests it handles, a classic ABBA deadlock can occur. This results in that django never boots, and that NO requests are handled. Just restarting apache works, (hoping that the same deadlock does not appear again). description of deadlock {{{ thread A thread B request foo import foo.views.py import foo.models.py request 'bar' import bar.models.py (acquires import lock B) bootstraps django calls django.models.loading.get_apps() self.write_lock.acquire() (lock A!) load app apps . . . django.models.loading.get_apps() import bar.models.py (takes import lock. lock B!) self.write_lock.acquire() (lock A!) }}} NB: this exact deadlock was '''actually''' seen in stracktraces. This is not a guess. == How to reproduce == its hard to reproduce, you need high volume traffic + multithreaded deployement + a big enough application that different urls trigger compelete different code paths in your project. I am not able to produce a simple test project that demonstrates this problem We had a 100% reproducible setup, where the site would lock up (almost) every time when we touched wsgi.py during peak hours of the day. 1. When you have a projects that is not multithreaded deployed -> this problem does not happen 2. When your application has a limited set of urls/views so that all initial calls follow same code paths -> this problem does not happen 3. When application has low traffic (the very first request can finish, without other requests beeing made) -> this problem does not happen. == Workaround == We use the attached {{{DjangoWSGIHandler}}} wrapper as a workaround. That solved the problem 100 % for us. Its a drop-in replacement for {{{ DjangoWSGIHandler }}} and effectively disables multithreading the first few request (allowing django to bootstrap properly), and only afther the first requests act multitheading again. Please consider if this workaround should be applied to django itself. (Put the Lock HIGHER the chain, to the wsgi handler level) == Fix == Some init locking should be adding to {{{ DjangoWSGIHandler }}} around the complete request. django.core.handlers.wsgi.WSGIHandler() has such lock around middleware loading. {{{ #!python if self._request_middleware is None: self.initLock.acquire() }}} Something similar should be around the complete request, not just middleware loading. But only the very first request. " Bug closed Core (Other) 1.3 Normal fixed hjwp2@… Ready for checkin 0 0 0 0 0 0