Opened 12 years ago

Closed 12 years ago

Last modified 12 years ago

#17658 closed Bug (needsinfo)

Django wsgi.py segfaults Apache

Reported by: harel@… Owned by: nobody
Component: Uncategorized Version: 1.3
Severity: Normal Keywords: wsgi apache segfault
Cc: anssi.kaariainen@… Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Ramiro Morales)

After a very standard and minor release, our Apache/Nginx/Django/wsgi setup ceased to function with the only error "Premature end of script headers: django.wsgi" in the apache log.
There is no expat problem (checked with sample wsgi app with and without that module include). The server was working fine. The codebase was updated slightly and apache restarted when the horrors began. The codebase was rolled back and still the problem persists.
Below are my wsgi file, apache config and a dump from running wsgi under the debuger.

I will have to rebuilt the server very soon, as the problem just won't go away. I am holding off in case anyone here finds this interesting and needs more information.

The system is Debian Squeeze, Apache/2.2.16 (Debian), Python 2.6.4, Django 1.3.1. modwsgi 3.3-2
All packages are installed via apt-get/pip. Another server with similar setup doesn't exhibit the problem.

wsgi file: 

import sys
import os
 
sys.path.append('/home/app/')
sys.path.append('/home/app/topmodule/')

os.environ['DJANGO_SETTINGS_MODULE'] = 'app.settings'
os.environ["CELERY_LOADER"] = "django"

import app.settings
import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

apache config: 

WSGIScriptAlias / /home/app/topmodule/django.wsgi
WSGIDaemonProcess app user=app group=app processes=6 threads=1
WSGIProcessGroup app 
Have also tried with : 
WSGIApplicationGroup %{GLOBAL}

Django Middleware:

'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'glowconsole.middleware.ViewNameMiddleware',
'django.middleware.gzip.GZipMiddleware'

Debugger dump when running apache2 -X with the pdb debugger wrapper (also here: ​http://paste.pocoo.org/show/546803/)

batman:/home/harel# /usr/sbin/apache2 -X
['/usr/local/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg', '/usr/local/lib/python26.zip', '/usr/local/lib/python2.6', '/usr/local/lib/python2.6/plat-linux2', '/usr/local/lib/python2.6/lib-tk', '/usr/local/lib/python2.6/lib-old', '/usr/local/lib/python2.6/lib-dynload', '/usr/local/lib/python2.6/site-packages', '/usr/local/lib/python2.6/site-packages/PIL', '/home/glowconsole/', '/home/glowconsole/glowconsole/', '/usr/local/lib/python2.6/site-packages/django']
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(240)__call__()
-> from django.conf import settings
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(244)__call__()
-> if self._request_middleware is None:
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(245)__call__()
-> self.initLock.acquire()
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(246)__call__()
-> try:
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(247)__call__()
-> try:
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(249)__call__()
-> if self._request_middleware is None:
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(250)__call__()
-> self.load_middleware()
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(256)__call__()
-> self.initLock.release()
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(258)__call__()
-> set_script_prefix(base.get_script_name(environ))
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(259)__call__()
-> signals.request_started.send(sender=self.__class__)
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(260)__call__()
-> try:
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(261)__call__()
-> try:
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(262)__call__()
-> request = self.request_class(environ)
(Pdb) n
> /usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(272)__call__()
-> response = self.get_response(request)
(Pdb) l
267  	                        'status_code': 400,
268  	                    }
269  	                )
270  	                response = http.HttpResponseBadRequest()
271  	            else:
272  ->	                response = self.get_response(request)
273  	        finally:
274  	            signals.request_finished.send(sender=self.__class__)
275  	
276  	        try:
277  	            status_text = STATUS_CODE_TEXT[response.status_code]
(Pdb) n
Segmentation fault

Some var dumps from that debugger session (also here: ​http://paste.pocoo.org/show/546846/):

(Pdb) request
<WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{'csrftoken': '6f41c55c934085b286fccb834e910643',
 'sessionid': '58a6595013bf69a456a532b318446315',
 'ys-account_id': 'n%3A585',
 'ys-ad_preview_window-1332956': 'o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272',
 'ys-client_id': 'n%3A202',
 'ys-title': 's%3ASlots%20Farm%20-CA%205'},
META:{'DOCUMENT_ROOT': '/etc/apache2/htdocs',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6',
 'HTTP_CACHE_CONTROL': 'max-age=0',
 'HTTP_CONNECTION': 'close',
 'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315',
 'HTTP_HOST': 'batman:8100',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11',
 'HTTP_X_FORWARDED_FOR': '217.138.9.130',
 'PATH_INFO': u'/',
 'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '81.94.195.18',
 'REMOTE_PORT': '54800',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/',
 'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi',
 'SCRIPT_NAME': u'',
 'SERVER_ADDR': '81.94.195.18',
 'SERVER_ADMIN': 'harel@xxxxxx.com',
 'SERVER_NAME': 'batman',
 'SERVER_PORT': '8100',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n',
 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)',
 'mod_wsgi.application_group': 'theglowmachine.com:8100|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': 'batman',
 'mod_wsgi.listener_port': '8100',
 'mod_wsgi.process_group': '',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>,
 'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

(Pdb) locals()

{'start_response': <built-in method start_response of mod_wsgi.Adapter object at 0x7f4f87277990>, 'request': <WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{'csrftoken': '6f41c55c934085b286fccb834e910643',
 'sessionid': '58a6595013bf69a456a532b318446315',
 'ys-account_id': 'n%3A585',
 'ys-ad_preview_window-1332956': 'o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272',
 'ys-client_id': 'n%3A202',
 'ys-title': 's%3ASlots%20Farm%20-CA%205'},
META:{'DOCUMENT_ROOT': '/etc/apache2/htdocs',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6',
 'HTTP_CACHE_CONTROL': 'max-age=0',
 'HTTP_CONNECTION': 'close',
 'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315',
 'HTTP_HOST': 'batman:8100',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11',
 'HTTP_X_FORWARDED_FOR': '217.138.9.130',
 'PATH_INFO': u'/',
 'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '81.94.195.18',
 'REMOTE_PORT': '54800',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/',
 'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi',
 'SCRIPT_NAME': u'',
 'SERVER_ADDR': '81.94.195.18',
 'SERVER_ADMIN': 'harel@thisisglow.com',
 'SERVER_NAME': 'batman',
 'SERVER_PORT': '8100',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n',
 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)',
 'mod_wsgi.application_group': 'theglowmachine.com:8100|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': 'batman',
 'mod_wsgi.listener_port': '8100',
 'mod_wsgi.process_group': '',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>,
 'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http', 
 'wsgi.version': (1, 1)}>, 'self': <django.core.handlers.wsgi.WSGIHandler object at 0x7f4f886a7d90>, 'environ': {'mod_wsgi.listener_port': '8100', 'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315', 'mod_wsgi.listener_host': 'batman', 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)', 'SCRIPT_NAME': u'', 'mod_wsgi.handler_script': '', 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n', 'REQUEST_METHOD': 'GET', 'PATH_INFO': u'/', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': '', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11', 'HTTP_CONNECTION': 'close', 'SERVER_NAME': 'batman', 'REMOTE_ADDR': '81.94.195.18', 'mod_wsgi.request_handler': 'wsgi-script', 'wsgi.url_scheme': 'http', 'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/', 'SERVER_PORT': '8100', 'wsgi.multiprocess': True, 'mod_wsgi.input_chunked': '0', 'SERVER_ADDR': '81.94.195.18', 'DOCUMENT_ROOT': '/etc/apache2/htdocs', 'mod_wsgi.process_group': '', 'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi', 'SERVER_ADMIN': 'harel@thisisglow.com', 'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>, 'HTTP_HOST': 'batman:8100', 'wsgi.multithread': True, 'mod_wsgi.callable_object': 'application', 'HTTP_CACHE_CONTROL': 'max-age=0', 'REQUEST_URI': '/', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'wsgi.version': (1, 1), 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_X_FORWARDED_FOR': '217.138.9.130', 'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>, 'REMOTE_PORT': '54800', 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6', 'mod_wsgi.version': (3, 3), 'wsgi.run_once': False, 'mod_wsgi.application_group': 'theglowmachine.com:8100|', 'mod_wsgi.script_reloading': '1', 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>, 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch'}, 'settings': <django.conf.LazySettings object at 0x7f4f721f6a50>}

Change History (7)

comment:1 by anonymous, 12 years ago

apologies for not using wiki formatting. noticed too late after posting and cannot edit

comment:2 by Ramiro Morales, 12 years ago

Description: modified (diff)

comment:3 by Anssi Kääriäinen, 12 years ago

Cc: anssi.kaariainen@… added

I really wonder if this is a bug in Django. All that can be concluded from the description (especially the "works on another server" part) suggests that there is something wrong with the server. Another hint is that upgrading and then downgrading and still getting this issue suggest that the problem is somewhere else.

Maybe this should be closed as needsinfo? I don't know much about wsgi, so I don't feel confident doing that.

comment:4 by anonymous, 12 years ago

Nothing really changed on the server though. Some python code was replaced with a slightly different version of same python code, then reverted back to the original. I do agree it is something on the server though, but a segfault like that with minimal information from wsgi.py prevents me from getting any way of debugging it.

in reply to:  4 comment:5 by Carl Meyer, 12 years ago

Resolution: → needsinfo
Status: new → closed

The segfault is on the "get_response" method call, which executes the entire request/response cycle, so the actual line of code that triggers the segfault could be anywhere, including in your own codebase. In any case, a segfault indicates a bug in C code (Apache, mod_wsgi, or Python itself), not in Django; it's theoretically possible that Django could trigger such a bug, and might be able to work around it, but that would require first knowing where and what the actual bug is.

Closing needsinfo, as there is no evidence of a bug in Django here.

comment:6 by Łukasz Rekucki, 12 years ago

I can only add that you should try debugging it with GDB. Here is a link to mod_wsgi docs:

​http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB.

comment:7 by anonymous, 12 years ago

I did try GDB but all I could get out of it is what looked like memory addresses which meant nothing to me. I'll rebuild the server and take it from there.

Note: See TracTickets for help on using tickets.
Back to Top