__iter__ on a QuerySet doesn't return an efficient iterator

[kenneth.arnold@…]

For a loop such as:

for item in Model.objects.all():
  print item # or whatever

the first line eats up all system memory for sufficiently large Models.

Replacing it with Model.objects.all().iterator() improves the situation a lot. Why doesn't iter return iterator()?

(I see that iter can use a cache. Is there a reason not to compute the cache on-the-fly?)

[Malcolm Tredinnick]

Since we cache the results anyway, this is all a bit of a non-event (because the memory will be used up anyway as you access the results) unless you're being very lazy with your queries and selecting way more information than you're using.

It's being changed in queryset-refactor in any case, for other reasons, so this ticket will be closed when that branch is merged.

[Malcolm Tredinnick]

(In [7030]) queryset-refactor: Converted the queryset iterator to be a real iterator and
only populate the result cache on demand. We actually populate the result cache
100 elements at a time, rather than one at a time for efficiency, but this is a
real win when the resultset contains 10,000 objects for example.

This also provides an efficient boolean (nonzero) test that doesn't use up
a lot of memory if you don't read all the results.

Refs #2430, #5987.

[Malcolm Tredinnick]

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658