Opened 8 years ago

Closed 7 years ago

Last modified 13 months ago

#3711 closed (fixed)

document that connection.queries will fill itself until system runs out of memory, and describe work-around

Reported by: sdistefano Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: connection.queries dict memory
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: yes
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

I found this when running a script which made thousonds of queries looping trough objects (not made to be efficient).
In any case, I think this could also happen just by running the server with DEBUG=True for an extended period of time.
This rendered my computer unusable (reboot required)

Attachments (1)

backends.diff (585 bytes) - added by Simon G. <dev@…> 8 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 8 years ago by Simon G. <dev@…>

  • Has patch set
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Accepted

Quick and dirty patch for this to delete older queries once it goes above (an arbitrarily chosen) number of queries, but I'm not sure that this is the best approach.

Changed 8 years ago by Simon G. <dev@…>

comment:2 Changed 8 years ago by ubernostrum

The query-logging behavior under DEBUG is documented and intended, and the potential for memory usage is also documented, so I'm not entirely certain there's an issue to be fixed here.

comment:3 Changed 8 years ago by Simon G. <dev@…>

This is true, but I don't think Django should take down the computer. However - how *many* queries do you need to do before this happens - we must be talking thousands of them.

comment:4 Changed 8 years ago by SmileyChris

I agree with Simon. And it his "quick and dirty" patch looks good enough. Perhaps change it to len(self.db.queries) > 1000 just for safety? :D

comment:5 Changed 8 years ago by PhiR

  • Needs documentation set
  • Triage Stage changed from Accepted to Design decision needed

Some people rely on having all the queries in memory to do some stats (like duplicates and stuff). Maybe we should add a security but hardcoding so low a value seems very restrictive.

comment:6 Changed 8 years ago by ubernostrum

  • Resolution set to wontfix
  • Status changed from new to closed

I'm going to wontfix per the above comment: we provide clear warnings that queries are logged in memory under DEBUG, and it's easy enough to manually flush it if that's causing you problems.

comment:7 Changed 8 years ago by rokclimb15@…

Does anyone know where this is documented? I ran into this issue recently, and my workaround was to set DEBUG to False, but I would love to know how to flush the query log from memory. I couldn't find any clues in the db backend code.

comment:8 Changed 8 years ago by Simon G <dev@…>

  • Has patch unset
  • Patch needs improvement unset
  • Resolution wontfix deleted
  • Status changed from closed to reopened
  • Summary changed from connection.queries will fill itself until system runs out of memory to document that connection.queries will fill itself until system runs out of memory, and describe work-around
  • Triage Stage changed from Design decision needed to Accepted

comment:9 Changed 7 years ago by Karen Tracey <kmtracey@…>

#6734 was opened with another proposal to cap connection.queries. I closed it as a duplicate of this.

comment:10 Changed 7 years ago by ubernostrum

  • Resolution set to wontfix
  • Status changed from reopened to closed

Guys, this is documented in the FAQ, and both the official docs and several people referencing them show up on the very first page of results for a Google search on "django memory leak"; if people already aren't doing the basic legwork to find the answer, it's unlikely they're looking at the docs at all, and so adding more documentation won't help those people in the least.

Marking wontfix again. Take it to django-dev if you have a convincing argument for why this should change, since ticket trackers suck as a discussion medium.

comment:11 Changed 13 months ago by Aymeric Augustin <aymeric.augustin@…>

  • Resolution changed from wontfix to fixed

In cfcca7ccce3dc527d16757ff6dc978e50c4a2e61:

Fixed #3711, #6734, #12581 -- Bounded connection.queries.

Prevented unlimited memory consumption when running background tasks
with DEBUG=True.

Thanks Rob, Alex, Baptiste, and others.

Note: See TracTickets for help on using tickets.
Back to Top