Opened 11 years ago

Closed 10 years ago

Last modified 3 years ago

#3711 closed (fixed)

document that connection.queries will fill itself until system runs out of memory, and describe work-around

Reported by: sdistefano Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: connection.queries dict memory
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: yes
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

I found this when running a script which made thousonds of queries looping trough objects (not made to be efficient).
In any case, I think this could also happen just by running the server with DEBUG=True for an extended period of time.
This rendered my computer unusable (reboot required)

Attachments (1)

backends.diff (585 bytes) - added by Simon G. <dev@…> 11 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 11 years ago by Simon G. <dev@…>

Has patch: set
Patch needs improvement: set
Triage Stage: UnreviewedAccepted

Quick and dirty patch for this to delete older queries once it goes above (an arbitrarily chosen) number of queries, but I'm not sure that this is the best approach.

Changed 11 years ago by Simon G. <dev@…>

Attachment: backends.diff added

comment:2 Changed 11 years ago by James Bennett

The query-logging behavior under DEBUG is documented and intended, and the potential for memory usage is also documented, so I'm not entirely certain there's an issue to be fixed here.

comment:3 Changed 11 years ago by Simon G. <dev@…>

This is true, but I don't think Django should take down the computer. However - how *many* queries do you need to do before this happens - we must be talking thousands of them.

comment:4 Changed 11 years ago by Chris Beaven

I agree with Simon. And it his "quick and dirty" patch looks good enough. Perhaps change it to len(self.db.queries) > 1000 just for safety? :D

comment:5 Changed 10 years ago by Philippe Raoult

Needs documentation: set
Triage Stage: AcceptedDesign decision needed

Some people rely on having all the queries in memory to do some stats (like duplicates and stuff). Maybe we should add a security but hardcoding so low a value seems very restrictive.

comment:6 Changed 10 years ago by James Bennett

Resolution: wontfix
Status: newclosed

I'm going to wontfix per the above comment: we provide clear warnings that queries are logged in memory under DEBUG, and it's easy enough to manually flush it if that's causing you problems.

comment:7 Changed 10 years ago by rokclimb15@…

Does anyone know where this is documented? I ran into this issue recently, and my workaround was to set DEBUG to False, but I would love to know how to flush the query log from memory. I couldn't find any clues in the db backend code.

comment:8 Changed 10 years ago by Simon G <dev@…>

Has patch: unset
Patch needs improvement: unset
Resolution: wontfix
Status: closedreopened
Summary: connection.queries will fill itself until system runs out of memorydocument that connection.queries will fill itself until system runs out of memory, and describe work-around
Triage Stage: Design decision neededAccepted

comment:9 Changed 10 years ago by Karen Tracey <kmtracey@…>

#6734 was opened with another proposal to cap connection.queries. I closed it as a duplicate of this.

comment:10 Changed 10 years ago by James Bennett

Resolution: wontfix
Status: reopenedclosed

Guys, this is documented in the FAQ, and both the official docs and several people referencing them show up on the very first page of results for a Google search on "django memory leak"; if people already aren't doing the basic legwork to find the answer, it's unlikely they're looking at the docs at all, and so adding more documentation won't help those people in the least.

Marking wontfix again. Take it to django-dev if you have a convincing argument for why this should change, since ticket trackers suck as a discussion medium.

comment:11 Changed 3 years ago by Aymeric Augustin <aymeric.augustin@…>

Resolution: wontfixfixed

In cfcca7ccce3dc527d16757ff6dc978e50c4a2e61:

Fixed #3711, #6734, #12581 -- Bounded connection.queries.

Prevented unlimited memory consumption when running background tasks
with DEBUG=True.

Thanks Rob, Alex, Baptiste, and others.

Note: See TracTickets for help on using tickets.
Back to Top