Opened 16 years ago

Closed 16 years ago

#5987 closed (fixed)

__iter__ on a QuerySet doesn't return an efficient iterator

Reported by: kenneth.arnold@… Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Keywords: iterator __iter__ qs-rf-fixed
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

For a loop such as:

for item in Model.objects.all():
  print item # or whatever

the first line eats up all system memory for sufficiently large Models.

Replacing it with Model.objects.all().iterator() improves the situation a lot. Why doesn't iter return iterator()?

(I see that iter can use a cache. Is there a reason not to compute the cache on-the-fly?)

Change History (4)

comment:1 by Malcolm Tredinnick, 16 years ago

Keywords: qs-rf added
Triage Stage: UnreviewedAccepted

Since we cache the results anyway, this is all a bit of a non-event (because the memory will be used up anyway as you access the results) unless you're being very lazy with your queries and selecting way more information than you're using.

It's being changed in queryset-refactor in any case, for other reasons, so this ticket will be closed when that branch is merged.

comment:2 by Malcolm Tredinnick, 16 years ago

(In [7030]) queryset-refactor: Converted the queryset iterator to be a real iterator and
only populate the result cache on demand. We actually populate the result cache
100 elements at a time, rather than one at a time for efficiency, but this is a
real win when the resultset contains 10,000 objects for example.

This also provides an efficient boolean (nonzero) test that doesn't use up
a lot of memory if you don't read all the results.

Refs #2430, #5987.

comment:3 by Malcolm Tredinnick, 16 years ago

Keywords: qs-rf-fixed added; qs-rf removed

comment:4 by Malcolm Tredinnick, 16 years ago

Resolution: fixed
Status: newclosed

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658

Note: See TracTickets for help on using tickets.
Back to Top