Opened 17 years ago

Closed 16 years ago

Last modified 10 years ago

#3711 closed (fixed)

document that connection.queries will fill itself until system runs out of memory, and describe work-around

Reported by: sdistefano Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Keywords: connection.queries dict memory
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: yes
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I found this when running a script which made thousonds of queries looping trough objects (not made to be efficient).
In any case, I think this could also happen just by running the server with DEBUG=True for an extended period of time.
This rendered my computer unusable (reboot required)

Attachments (1)

backends.diff (585 bytes ) - added by Simon G. <dev@…> 17 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 by Simon G. <dev@…>, 17 years ago

Has patch: set
Patch needs improvement: set
Triage Stage: UnreviewedAccepted

Quick and dirty patch for this to delete older queries once it goes above (an arbitrarily chosen) number of queries, but I'm not sure that this is the best approach.

by Simon G. <dev@…>, 17 years ago

Attachment: backends.diff added

comment:2 by James Bennett, 17 years ago

The query-logging behavior under DEBUG is documented and intended, and the potential for memory usage is also documented, so I'm not entirely certain there's an issue to be fixed here.

comment:3 by Simon G. <dev@…>, 17 years ago

This is true, but I don't think Django should take down the computer. However - how *many* queries do you need to do before this happens - we must be talking thousands of them.

comment:4 by Chris Beaven, 17 years ago

I agree with Simon. And it his "quick and dirty" patch looks good enough. Perhaps change it to len(self.db.queries) > 1000 just for safety? :D

comment:5 by Philippe Raoult, 17 years ago

Needs documentation: set
Triage Stage: AcceptedDesign decision needed

Some people rely on having all the queries in memory to do some stats (like duplicates and stuff). Maybe we should add a security but hardcoding so low a value seems very restrictive.

comment:6 by James Bennett, 17 years ago

Resolution: wontfix
Status: newclosed

I'm going to wontfix per the above comment: we provide clear warnings that queries are logged in memory under DEBUG, and it's easy enough to manually flush it if that's causing you problems.

comment:7 by rokclimb15@…, 16 years ago

Does anyone know where this is documented? I ran into this issue recently, and my workaround was to set DEBUG to False, but I would love to know how to flush the query log from memory. I couldn't find any clues in the db backend code.

comment:8 by Simon G <dev@…>, 16 years ago

Has patch: unset
Patch needs improvement: unset
Resolution: wontfix
Status: closedreopened
Summary: connection.queries will fill itself until system runs out of memorydocument that connection.queries will fill itself until system runs out of memory, and describe work-around
Triage Stage: Design decision neededAccepted

comment:9 by Karen Tracey <kmtracey@…>, 16 years ago

#6734 was opened with another proposal to cap connection.queries. I closed it as a duplicate of this.

comment:10 by James Bennett, 16 years ago

Resolution: wontfix
Status: reopenedclosed

Guys, this is documented in the FAQ, and both the official docs and several people referencing them show up on the very first page of results for a Google search on "django memory leak"; if people already aren't doing the basic legwork to find the answer, it's unlikely they're looking at the docs at all, and so adding more documentation won't help those people in the least.

Marking wontfix again. Take it to django-dev if you have a convincing argument for why this should change, since ticket trackers suck as a discussion medium.

comment:11 by Aymeric Augustin <aymeric.augustin@…>, 10 years ago

Resolution: wontfixfixed

In cfcca7ccce3dc527d16757ff6dc978e50c4a2e61:

Fixed #3711, #6734, #12581 -- Bounded connection.queries.

Prevented unlimited memory consumption when running background tasks
with DEBUG=True.

Thanks Rob, Alex, Baptiste, and others.

Note: See TracTickets for help on using tickets.
Back to Top