#13214 closed (invalid)
Random OperationalError when using FastCGI (+ possible solutions)
Reported by: | Henrique C. Alves | Owned by: | nobody |
---|---|---|---|
Component: | Core (Other) | Version: | 1.1 |
Severity: | Keywords: | ||
Cc: | hcarvalhoalves@… | Triage Stage: | Unreviewed |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Possible solution: http://groups.google.com/group/django-users/browse_thread/thread/2c7421cdb9b99e48
Until recently I was curious to test
this on Django 1.1.1. Will this
exception be thrown again... surprise,
there it was again. It took me some
time to debug this, helpful hint was
that it only shows when (pre)forking.
So for those who getting randomly
those exceptions, I can say... fix
your code :) Ok.. seriously, there
are always few ways of doing this, so
let me firs explain where is a
problem first. If you access database
when any of your modules will import
as, e.g. reading configuration from
database then you will get this error.
When your fastcgi-prefork application
starts, first it imports all modules,
and only after this forks children.
If you have established db connection
during import all children processes
will have an exact copy of that
object. This connection is being
closed at the end of request phase
(request_finished signal). So first
child which will be called to process
your request, will close this
connection. But what will happen to
the rest of the child processes? They
will believe that they have open and
presumably working connection to the
db, so any db operation will cause an
exception. Why this is not showing in
threaded execution model? I suppose
because threads are using same object
and know when any other thread is
closing connection. How to fix this?
Best way is to fix your code... but
this can be difficult sometimes.
Other option, in my opinion quite
clean, is to write somewhere in your
application small piece of code:
from django.db import connection from django.core import signals def close_connection(**kwargs): connection.close() signals.request_started.connect(close_connection)
Not ideal thought, connecting twice to the DB is a workaround at best.
Possible solution: using connection pooling (pgpool, pgbouncer), so you have DB connections pooled and stable, and handed fast to your FCGI daemons.
The problem is that this triggers another bug, psycopg2 raising an *InterfaceError* because it's trying to disconnect twice (pgbouncer already handled this).
Now the culprit is Django signal *request_finished* triggering *connection.close()*, and failing loud even if it was already disconnected. I don't think this behavior is desired, as if the request already finished, we don't care about the DB connection anymore. A patch for correcting this should be simple.
The relevant traceback:
/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/core/handlers/wsgi.py in __call__(self=<django.core.handlers.wsgi.WSGIHandler object at 0x24fb210>, environ={'AUTH_TYPE': 'Basic', 'DOCUMENT_ROOT': '/storage/test', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTPS': 'off', 'HTTP_ACCEPT': 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', 'HTTP_AUTHORIZATION': 'Basic dGVzdGU6c3VjZXNzbw==', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_COOKIE': '__utma=175602209.1371964931.1269354495.126938948...none); sessionid=a1990f0d8d32c78a285489586c510e8c', 'HTTP_HOST': 'www.rede-colibri.com', ...}, start_response=<function start_response at 0x24f87d0>) 246 response = self.apply_response_fixes(request, response) 247 finally: 248 signals.request_finished.send(sender=self.__class__) 249 250 try: global signals = <module 'django.core.signals' from '/usr/local/l.../Django-1.1.1-py2.6.egg/django/core/signals.pyc'>, signals.request_finished = <django.dispatch.dispatcher.Signal object at 0x1975710>, signals.request_finished.send = <bound method Signal.send of <django.dispatch.dispatcher.Signal object at 0x1975710>>, sender undefined, self = <django.core.handlers.wsgi.WSGIHandler object at 0x24fb210>, self.__class__ = <class 'django.core.handlers.wsgi.WSGIHandler'> /usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/dispatch/dispatcher.py in send(self=<django.dispatch.dispatcher.Signal object at 0x1975710>, sender=<class 'django.core.handlers.wsgi.WSGIHandler'>, **named={}) 164 165 for receiver in self._live_receivers(_make_id(sender)): 166 response = receiver(signal=self, sender=sender, **named) 167 responses.append((receiver, response)) 168 return responses response undefined, receiver = <function close_connection at 0x197b050>, signal undefined, self = <django.dispatch.dispatcher.Signal object at 0x1975710>, sender = <class 'django.core.handlers.wsgi.WSGIHandler'>, named = {} /usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/__init__.py in close_connection(**kwargs={'sender': <class 'django.core.handlers.wsgi.WSGIHandler'>, 'signal': <django.dispatch.dispatcher.Signal object at 0x1975710>}) 63 # when a Django request is finished. 64 def close_connection(**kwargs): 65 connection.close() 66 signals.request_finished.connect(close_connection) 67 global connection = <django.db.backends.postgresql_psycopg2.base.DatabaseWrapper object at 0x17b14c8>, connection.close = <bound method DatabaseWrapper.close of <django.d...ycopg2.base.DatabaseWrapper object at 0x17b14c8>> /usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/backends/__init__.py in close(self=<django.db.backends.postgresql_psycopg2.base.DatabaseWrapper object at 0x17b14c8>) 74 def close(self): 75 if self.connection is not None: 76 self.connection.close() 77 self.connection = None 78 self = <django.db.backends.postgresql_psycopg2.base.DatabaseWrapper object at 0x17b14c8>, self.connection = <connection object at 0x1f80870; dsn: 'dbname=co...st=127.0.0.1 port=6432 user=postgres', closed: 2>, self.connection.close = <built-in method close of psycopg2._psycopg.connection object at 0x1f80870>
Exception handling here could add more leniency:
**/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/__init__.py** 63 # when a Django request is finished. 64 def close_connection(**kwargs): 65 connection.close() 66 signals.request_finished.connect(close_connection)
Or it could be handled better on psycopg2, so to not throw fatal errors if all we're trying to do is disconnect and it already is:
**/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/backends/__init__.py** 74 def close(self): 75 if self.connection is not None: 76 self.connection.close() 77 self.connection = None
Other than that, I'm short on ideas. Not sure if this is supposed to be fixed on 1.2, maybe multi-db support refactored things and dealt with this issue. I'll try testing with a 1.2 enviro.
Change History (5)
comment:1 by , 15 years ago
Cc: | added |
---|---|
Component: | Uncategorized → Core framework |
comment:2 by , 15 years ago
comment:3 by , 15 years ago
Working in threaded mode (./manage.py runfcgi method=threaded) seems to solve the problem, at least I'm unable to reproduce anymore. For the record I'm using connection pooling. Is it safe to deploy Django in threaded mode? Doesn't it impact performance because of the GIL?
If there's a solution for using in prefork mode too, would be invaluable.
For others struggling with this like me, my setup:
- Cherokee Web Server (0.44.19)
- PostgreSQL 8.3
- pgbouncer (set for 'client' reset mode)
- Django 1.1
Daemon command line:
python manage.py runfcgi protocol=fcgi method=threaded
comment:4 by , 15 years ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
Ok, I've read this ticket a couple of times now, and I have no clue what's going on. I don't mean I'm having problems working out the cause of the bug - I'm having problems *understanding what the bug is*.
This report contains a 3 (maybe 4) "solutions". It contains links to mailing lists. It contains snippets of rambling run-on sentences. It contains multiple page stack traces. It contains speculation that the problem might be fixed in 1.2/trunk.
The one thing it doesn't contain is 'a description of the problem, and the conditions under which it will be experienced'. The only hints we have are in the ticket title, and they're not especially illuminating.
I'm closing this one Invalid as a procedural "No. Seriously. WTF". Assuming that you have actually found a legitimate problem (and it's impossible to tell because of the lack of details), nobody else should be forced to spelunk their way through this sort of sprawling ramble to work out what the hell this report is actually reporting.
If you think this is actually a genuine issue, start again with a clean ticket. The ticket description should be a 'clear, concise description of the problem, and the conditions under which it can be observed'. It should contain 'zero' discussion of proposed solutions. That's what the ticket comments are for.
comment:5 by , 15 years ago
I was just trying to provide links and discussion happening to other pages where people are discussing this issue and consolidate in one ticket for people that, like me, searched the tickets for something like this and found nothing.
No wonder no one bothers filling a ticket here, considering the warm welcome I received. Thanks.
Filling a new ticket now.
In fact, pooling won't help at all. I tested a bit more, and found the offending code in my own project.
I have this middleware:
On daemon startup, Site.objects.get_current() causes the query to miss the cache and hit the database:
Does that mean that is impossible to query the database from a middleware when using FastCGI + prefork? Makes no sense, the deployment method shouldn't trigger bugs. Looking forward for any possible solution or workaround.