Opened 3 years ago

Closed 3 years ago

#27066 closed Uncategorized (needsinfo)

Possible Memory leak while evaluating a QuerySet

Reported by: Ramez Issac Owned by: nobody
Component: Database layer (models, ORM) Version: 1.8
Severity: Normal Keywords: queryset evaluation memory leak
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I discovered this as i have a 150K rows QuerySet,
I checked its existence by the not efficient

if queryset:
    pass

However, as soon as execution hits this line, memory consumption jumps big time and it never goes back to normal unless uwsgi process gets restarted.
This information has been noticed using htop showing the relevant uwsgi process.

Using query.exists() is working perfectly good.

Change History (6)

comment:1 Changed 3 years ago by Tim Graham

As the documentation says:

Testing a QuerySet in a boolean context, such as using bool(), or, and or an if statement, will cause the query to be executed. If there is at least one result, the QuerySet is True, otherwise False. If you only want to determine if at least one result exists (and don’t need the actual objects), it’s more efficient to use exists().

Can you provide more details if that doesn't address your concerns?

comment:2 Changed 3 years ago by Ramez Issac

Thank you dear time for your quick reply,

What i'm referring to here is when i made the mistake of using bool(queryset) , the server memory jumped and never returned back to normal even after the function exit.
Something is not right here.
The only way to bring the memory back to normal was by restarting the uwsgi process running the django instance.

It's not about that the line bool(queryset) takes longer time to execute, or is heavy on memory;
But this line have a long lasting side effect on server memory consumption.

comment:3 Changed 3 years ago by Tim Graham

I'm not intimately familiar with memory management in Python but unless you can provide details as to why, I'm doubtful that Django is at fault for this behavior. Maybe this post about Diagnosing Memory "Leaks" in Python helps.

comment:4 Changed 3 years ago by Claude Paroz

You don't run with DEBUG=True, right?

comment:5 Changed 3 years ago by Ramez Issac

@claudep Lol Nope ! :D

@Tim I'll review the document you attached and get back to you with more detailed/insightful information.

comment:6 Changed 3 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top