Opened 10 years ago

Closed 10 years ago

#23462 closed Bug (invalid)

Django 1.7 openning one database connection for each request

Reported by: Mauro Alexandre Owned by: nobody
Component: Database layer (models, ORM) Version: 1.7
Severity: Normal Keywords: mysql
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I use MariaDB database.

After updated my production environment from Django 1.5 to Django 1.7, i'm facing "Too many open connections".

Django opening one connection for each request. I tried input CONN_MAX_AGE explicit in settings but not solved. My debug settings is False.

For more information about environment or application, please request.

Change History (9)

comment:1 by Aymeric Augustin, 10 years ago

This might be related to my changes to connection management in Django 1.6.

However, I need more information to confirm.

1) Does your application hit the database for every request?

2) How many Django processes / threads do you run?

3) How many connections does your database server accept?

in reply to:  1 comment:2 by Mauro Alexandre, 10 years ago

Replying to aaugustin:

This might be related to my changes to connection management in Django 1.6.

However, I need more information to confirm.

1) Does your application hit the database for every request?

Yes, because in header of my system i have one function used for check versions, and another informations in database.

2) How many Django processes / threads do you run?

I use wsgi + apache2 with default configs.

3) How many connections does your database server accept?

Now, i changed my config file my.cnf for allow 1500 connections, because my enviroment is down. But for me, is normal about 300 maximum.

Another details:

I'm trying debug core form django, and I see something, in django/db/backends/init.py i put "print", to check about close.

Code highlighting:

  def close(self):
      """
      Closes the connection to the database.
      """
      print('1')
      self.validate_thread_sharing()
      # Don't call validate_no_atomic_block() to avoid making it difficult
      # to get rid of a connection in an invalid state. The next connect()
      # will reset the transaction state anyway.

      if self.closed_in_transaction or self.connection is None:
          print('2')
          return
      try:
          self._close()
          print('3')
      finally:
          print('4')
          if self.in_atomic_block:
              print('5')
              self.closed_in_transaction = True
              self.needs_rollback = True
          else:
              print('6')
              self.connection = None
      print('7')
      self.set_clean()

The aplication print, 1,3,4,6,7.

Last edited 10 years ago by Mauro Alexandre (previous) (diff)

comment:3 by Aymeric Augustin, 10 years ago

There are two reasons why you shouldn't see a difference between 1.5 and 1.6+:

  • If I understand correctly, you've set CONN_MAX_AGE to 0 -- the default; that's supposed to preserve the 1.5 behavior of opening and closing a database connection for each request.
  • Even if you've set CONN_MAX_AGE to something >0, since every request hits the database on your site, that shouldn't change much, unless you had many idle Apache/mod_wsgi workers.

Unless I missed something, your application was already opening one connection for each request, and it still does; there's no change.

I don't know what Apache's defaults are and they may depend on your hardware — I assume Apache is able to scale its processes depending on how many CPUs you have.

The sequence you've posted (1, 3, 4, 6, 7) looks correct. The connection is terminated properly.

I would suggest that you:

  • Review how many database connections you expect to need. That's the number of Apache/mod_wsgi processes x the number of threads per process, plus anything else that may connect to the database.
  • Configure the database to allow this number of connections, plus a small safety margin.
  • (if possible) Measure how many connections the database actually gets and figure out if it's consistent your Apache setp.
  • (optional) If you wish, enable persistent connections by setting CONN_MAX_AGE to something like 60, which should make your website faster.

comment:4 by Mauro Alexandre, 10 years ago

aaugustin, sorry about that.

I migrate from apache to nginx, with 3 workers only, and solved the problem.

You is correct about threads and mod_wsgi :)

But, in the end, this is a good experience, I learned a lot, including some of the django code.

Thank you !

comment:5 by Mauro Alexandre, 10 years ago

Resolution: invalid
Status: newclosed

comment:6 by Mauro Alexandre, 10 years ago

Resolution: invalid
Status: closednew

comment:7 by Mauro Alexandre, 10 years ago

The problem continues, i don't have more ideas.

comment:8 by Shai Berger, 10 years ago

Since you do see the connection being closed (according to your own debug info), I suspect that the problem is that you are opening more than one connection per request.

The common reason for this to happen is that the application opens a separate thread and does some database queries in it.

Another possibility is that you do something in a middleware's process_response() -- this could open another connection on the same thread after the "main" connection has been closed.

In summary - try to debug the opening of connections, not just the closing.

comment:9 by Aymeric Augustin, 10 years ago

Resolution: invalid
Status: newclosed

Since we don't have evidence of a bug in Django or ideas for improvements, I'm going to close this ticket.

Please see TicketClosingReasons/UseSupportChannels for ways to get help. Thanks!

Note: See TracTickets for help on using tickets.
Back to Top