Opened 9 years ago

Closed 7 years ago

#2430 closed defect (fixed)

QuerySet should be evaluated when used in boolean tests

Reported by: ubernostrum Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: normal Keywords: qs-rf-fixed
Cc: ferringb@…, harish.mallipeddi@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Currently, a QuerySet is not evaluated when being used in boolean contexts (e.g., doing if SomeModel.objects.all():); this leads to counterintuitive behavior, because it will always resolve to a True value regardless of whether it would return any objects when "actually" evaluated.

Adding a __nonzero__ method to QuerySet which does the evaluation would solve this.

Attachments (1)

nonzero_fix.diff (507 bytes) - added by anonymous 8 years ago.
QuerySet nonzero fix

Download all attachments as: .zip

Change History (14)

comment:1 Changed 9 years ago by mtredinnick

  • Status changed from new to assigned

comment:2 Changed 9 years ago by mir@…

I tried it out and this seems to have changed, at least under python 2.4. Can you test again, please?

>>> if models.Mailrule.objects.filter(quelle="aaaa"):
        print "boing"
>>> if models.Mailrule.objects.all():
...     print "yepp"
yepp

>>>

comment:3 Changed 9 years ago by mir@…

  • Resolution set to fixed
  • Status changed from assigned to closed

Well, then I count it as "has been fixed."

comment:4 Changed 9 years ago by mtredinnick

  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Triage Stage changed from Unreviewed to Accepted

This has not been fixed, because evaluating it in the above context calls len() to be called on the QuerySet. This is a very bad idea if you have, say, 100K rows in the result set (it reads them all into memory). We do need to implement a proper __nonzero___ method that only looks for the existence of one row without consuming it (and this is only mildly hard because the rowcount attribute on a database cursor in SQLite always returns -1).

Reopening. I'll fix this one day. It's something that is needed.

comment:5 Changed 9 years ago by Michael Radziej <mir@…>

Well ... what you describe is actually a different from what has been reported.

Anyway, I'd propose to turn this into a query like:

select exists(select * from ...)

Unfortunately, this might turn out to be backend specific, too. For example, oracle doesn't like select queries without a table, IIRC. With mysql and postgresql, the above syntax works.

comment:6 Changed 9 years ago by mtredinnick

Using an extra SQL query is not the way to go here; what if the query is time consuming?. We already have the result set. We just need to read the first row (in the cases where "rowcount" is not honoured on cursors) and if we get data back, cache it and remember to add it to future retrievals. A little fiddly, but not impossible and imposes no extra database load.

comment:7 Changed 8 years ago by anonymous

  • Cc ferringb@… added

Changed 8 years ago by anonymous

QuerySet nonzero fix

comment:8 Changed 8 years ago by mallipeddi

  • Has patch set

I just submitted a simple patch which essentially adds a nonzero method to the QuerySet class. Inside the nonzero method, I fetch the first item via getitem call. This is either answered from the result cache (if any) or would result in a SELECT query with the LIMIT set to 1.

comment:9 Changed 8 years ago by mallipeddi

  • Cc harish.mallipeddi@… added

comment:10 Changed 8 years ago by mtredinnick

  • Keywords qs-rf added

comment:11 Changed 8 years ago by mtredinnick

(In [7030]) queryset-refactor: Converted the queryset iterator to be a real iterator and
only populate the result cache on demand. We actually populate the result cache
100 elements at a time, rather than one at a time for efficiency, but this is a
real win when the resultset contains 10,000 objects for example.

This also provides an efficient boolean (nonzero) test that doesn't use up
a lot of memory if you don't read all the results.

Refs #2430, #5987.

comment:12 Changed 8 years ago by mtredinnick

  • Keywords qs-rf-fixed added; qs-rf removed

comment:13 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658

Note: See TracTickets for help on using tickets.
Back to Top