#17658 closed Bug (needsinfo)
Django wsgi.py segfaults Apache
Reported by: | Owned by: | nobody | |
---|---|---|---|
Component: | Uncategorized | Version: | 1.3 |
Severity: | Normal | Keywords: | wsgi apache segfault |
Cc: | anssi.kaariainen@… | Triage Stage: | Unreviewed |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
After a very standard and minor release, our Apache/Nginx/Django/wsgi
setup ceased to function with the only error "Premature end of script headers: django.wsgi" in the apache log.
There is no expat problem (checked with sample wsgi app with and without that module include). The server was working fine. The codebase was updated slightly and apache restarted when the horrors began. The codebase was rolled back and still the problem persists.
Below are my wsgi file, apache config and a dump from running wsgi under the debuger.
I will have to rebuilt the server very soon, as the problem just won't go away. I am holding off in case anyone here finds this interesting and needs more information.
The system is Debian Squeeze, Apache/2.2.16 (Debian), Python 2.6.4, Django 1.3.1. modwsgi 3.3-2
All packages are installed via apt-get/pip. Another server with similar setup doesn't exhibit the problem.
wsgi file:
import sys import os sys.path.append('/home/app/') sys.path.append('/home/app/topmodule/') os.environ['DJANGO_SETTINGS_MODULE'] = 'app.settings' os.environ["CELERY_LOADER"] = "django" import app.settings import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()
apache config:
WSGIScriptAlias / /home/app/topmodule/django.wsgi WSGIDaemonProcess app user=app group=app processes=6 threads=1 WSGIProcessGroup app Have also tried with : WSGIApplicationGroup %{GLOBAL}
Django Middleware:
'django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'glowconsole.middleware.ViewNameMiddleware', 'django.middleware.gzip.GZipMiddleware'
Debugger dump when running apache2 -X with the pdb debugger wrapper (also here: http://paste.pocoo.org/show/546803/)
batman:/home/harel# /usr/sbin/apache2 -X ['/usr/local/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg', '/usr/local/lib/python26.zip', '/usr/local/lib/python2.6', '/usr/local/lib/python2.6/plat-linux2', '/usr/local/lib/python2.6/lib-tk', '/usr/local/lib/python2.6/lib-old', '/usr/local/lib/python2.6/lib-dynload', '/usr/local/lib/python2.6/site-packages', '/usr/local/lib/python2.6/site-packages/PIL', '/home/glowconsole/', '/home/glowconsole/glowconsole/', '/usr/local/lib/python2.6/site-packages/django'] > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(240)__call__() -> from django.conf import settings (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(244)__call__() -> if self._request_middleware is None: (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(245)__call__() -> self.initLock.acquire() (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(246)__call__() -> try: (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(247)__call__() -> try: (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(249)__call__() -> if self._request_middleware is None: (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(250)__call__() -> self.load_middleware() (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(256)__call__() -> self.initLock.release() (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(258)__call__() -> set_script_prefix(base.get_script_name(environ)) (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(259)__call__() -> signals.request_started.send(sender=self.__class__) (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(260)__call__() -> try: (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(261)__call__() -> try: (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(262)__call__() -> request = self.request_class(environ) (Pdb) n > /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(272)__call__() -> response = self.get_response(request) (Pdb) l 267 'status_code': 400, 268 } 269 ) 270 response = http.HttpResponseBadRequest() 271 else: 272 -> response = self.get_response(request) 273 finally: 274 signals.request_finished.send(sender=self.__class__) 275 276 try: 277 status_text = STATUS_CODE_TEXT[response.status_code] (Pdb) n Segmentation fault
Some var dumps from that debugger session (also here: http://paste.pocoo.org/show/546846/):
(Pdb) request <WSGIRequest GET:<QueryDict: {}>, POST:<QueryDict: {}>, COOKIES:{'csrftoken': '6f41c55c934085b286fccb834e910643', 'sessionid': '58a6595013bf69a456a532b318446315', 'ys-account_id': 'n%3A585', 'ys-ad_preview_window-1332956': 'o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272', 'ys-client_id': 'n%3A202', 'ys-title': 's%3ASlots%20Farm%20-CA%205'}, META:{'DOCUMENT_ROOT': '/etc/apache2/htdocs', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch', 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6', 'HTTP_CACHE_CONTROL': 'max-age=0', 'HTTP_CONNECTION': 'close', 'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315', 'HTTP_HOST': 'batman:8100', 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11', 'HTTP_X_FORWARDED_FOR': '217.138.9.130', 'PATH_INFO': u'/', 'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/', 'QUERY_STRING': '', 'REMOTE_ADDR': '81.94.195.18', 'REMOTE_PORT': '54800', 'REQUEST_METHOD': 'GET', 'REQUEST_URI': '/', 'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi', 'SCRIPT_NAME': u'', 'SERVER_ADDR': '81.94.195.18', 'SERVER_ADMIN': 'harel@xxxxxx.com', 'SERVER_NAME': 'batman', 'SERVER_PORT': '8100', 'SERVER_PROTOCOL': 'HTTP/1.0', 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n', 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)', 'mod_wsgi.application_group': 'theglowmachine.com:8100|', 'mod_wsgi.callable_object': 'application', 'mod_wsgi.handler_script': '', 'mod_wsgi.input_chunked': '0', 'mod_wsgi.listener_host': 'batman', 'mod_wsgi.listener_port': '8100', 'mod_wsgi.process_group': '', 'mod_wsgi.request_handler': 'wsgi-script', 'mod_wsgi.script_reloading': '1', 'mod_wsgi.version': (3, 3), 'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>, 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>, 'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>, 'wsgi.multiprocess': True, 'wsgi.multithread': True, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 1)}>
(Pdb) locals()
{'start_response': <built-in method start_response of mod_wsgi.Adapter object at 0x7f4f87277990>, 'request': <WSGIRequest GET:<QueryDict: {}>, POST:<QueryDict: {}>, COOKIES:{'csrftoken': '6f41c55c934085b286fccb834e910643', 'sessionid': '58a6595013bf69a456a532b318446315', 'ys-account_id': 'n%3A585', 'ys-ad_preview_window-1332956': 'o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272', 'ys-client_id': 'n%3A202', 'ys-title': 's%3ASlots%20Farm%20-CA%205'}, META:{'DOCUMENT_ROOT': '/etc/apache2/htdocs', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch', 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6', 'HTTP_CACHE_CONTROL': 'max-age=0', 'HTTP_CONNECTION': 'close', 'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315', 'HTTP_HOST': 'batman:8100', 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11', 'HTTP_X_FORWARDED_FOR': '217.138.9.130', 'PATH_INFO': u'/', 'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/', 'QUERY_STRING': '', 'REMOTE_ADDR': '81.94.195.18', 'REMOTE_PORT': '54800', 'REQUEST_METHOD': 'GET', 'REQUEST_URI': '/', 'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi', 'SCRIPT_NAME': u'', 'SERVER_ADDR': '81.94.195.18', 'SERVER_ADMIN': 'harel@thisisglow.com', 'SERVER_NAME': 'batman', 'SERVER_PORT': '8100', 'SERVER_PROTOCOL': 'HTTP/1.0', 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n', 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)', 'mod_wsgi.application_group': 'theglowmachine.com:8100|', 'mod_wsgi.callable_object': 'application', 'mod_wsgi.handler_script': '', 'mod_wsgi.input_chunked': '0', 'mod_wsgi.listener_host': 'batman', 'mod_wsgi.listener_port': '8100', 'mod_wsgi.process_group': '', 'mod_wsgi.request_handler': 'wsgi-script', 'mod_wsgi.script_reloading': '1', 'mod_wsgi.version': (3, 3), 'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>, 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>, 'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>, 'wsgi.multiprocess': True, 'wsgi.multithread': True, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 1)}>, 'self': <django.core.handlers.wsgi.WSGIHandler object at 0x7f4f886a7d90>, 'environ': {'mod_wsgi.listener_port': '8100', 'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315', 'mod_wsgi.listener_host': 'batman', 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)', 'SCRIPT_NAME': u'', 'mod_wsgi.handler_script': '', 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n', 'REQUEST_METHOD': 'GET', 'PATH_INFO': u'/', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': '', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11', 'HTTP_CONNECTION': 'close', 'SERVER_NAME': 'batman', 'REMOTE_ADDR': '81.94.195.18', 'mod_wsgi.request_handler': 'wsgi-script', 'wsgi.url_scheme': 'http', 'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/', 'SERVER_PORT': '8100', 'wsgi.multiprocess': True, 'mod_wsgi.input_chunked': '0', 'SERVER_ADDR': '81.94.195.18', 'DOCUMENT_ROOT': '/etc/apache2/htdocs', 'mod_wsgi.process_group': '', 'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi', 'SERVER_ADMIN': 'harel@thisisglow.com', 'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>, 'HTTP_HOST': 'batman:8100', 'wsgi.multithread': True, 'mod_wsgi.callable_object': 'application', 'HTTP_CACHE_CONTROL': 'max-age=0', 'REQUEST_URI': '/', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'wsgi.version': (1, 1), 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_X_FORWARDED_FOR': '217.138.9.130', 'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>, 'REMOTE_PORT': '54800', 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6', 'mod_wsgi.version': (3, 3), 'wsgi.run_once': False, 'mod_wsgi.application_group': 'theglowmachine.com:8100|', 'mod_wsgi.script_reloading': '1', 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>, 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch'}, 'settings': <django.conf.LazySettings object at 0x7f4f721f6a50>}
Change History (7)
comment:1 by , 13 years ago
comment:2 by , 13 years ago
Description: | modified (diff) |
---|
comment:3 by , 13 years ago
Cc: | added |
---|
I really wonder if this is a bug in Django. All that can be concluded from the description (especially the "works on another server" part) suggests that there is something wrong with the server. Another hint is that upgrading and then downgrading and still getting this issue suggest that the problem is somewhere else.
Maybe this should be closed as needsinfo? I don't know much about wsgi, so I don't feel confident doing that.
follow-up: 5 comment:4 by , 13 years ago
Nothing really changed on the server though. Some python code was replaced with a slightly different version of same python code, then reverted back to the original. I do agree it is something on the server though, but a segfault like that with minimal information from wsgi.py prevents me from getting any way of debugging it.
comment:5 by , 13 years ago
Resolution: | → needsinfo |
---|---|
Status: | new → closed |
The segfault is on the "get_response" method call, which executes the entire request/response cycle, so the actual line of code that triggers the segfault could be anywhere, including in your own codebase. In any case, a segfault indicates a bug in C code (Apache, mod_wsgi, or Python itself), not in Django; it's theoretically possible that Django could trigger such a bug, and might be able to work around it, but that would require first knowing where and what the actual bug is.
Closing needsinfo, as there is no evidence of a bug in Django here.
comment:6 by , 13 years ago
I can only add that you should try debugging it with GDB. Here is a link to mod_wsgi docs:
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB.
comment:7 by , 13 years ago
I did try GDB but all I could get out of it is what looked like memory addresses which meant nothing to me. I'll rebuild the server and take it from there.
apologies for not using wiki formatting. noticed too late after posting and cannot edit