Django

Code

Ticket #5987 (closed: fixed)

Opened 6 months ago

Last modified 3 weeks ago

__iter__ on a QuerySet doesn't return an efficient iterator

Reported by: kenneth.arnold@gmail.com Assigned to: nobody
Component: Database wrapper Version: SVN
Keywords: iterator __iter__ qs-rf-fixed Cc:
Triage Stage: Accepted Has patch: 0
Needs documentation: 0 Needs tests: 0
Patch needs improvement: 0

Description

For a loop such as:

for item in Model.objects.all():
  print item # or whatever

the first line eats up all system memory for sufficiently large Models.

Replacing it with Model.objects.all().iterator() improves the situation a lot. Why doesn't iter return iterator()?

(I see that iter can use a cache. Is there a reason not to compute the cache on-the-fly?)

Attachments

Change History

11/19/07 18:17:03 changed by mtredinnick

  • keywords changed from iterator __iter__ to iterator __iter__ qs-rf.
  • needs_better_patch changed.
  • stage changed from Unreviewed to Accepted.
  • needs_tests changed.
  • needs_docs changed.

Since we cache the results anyway, this is all a bit of a non-event (because the memory will be used up anyway as you access the results) unless you're being very lazy with your queries and selecting way more information than you're using.

It's being changed in queryset-refactor in any case, for other reasons, so this ticket will be closed when that branch is merged.

01/26/08 07:23:55 changed by mtredinnick

(In [7030]) queryset-refactor: Converted the queryset iterator to be a real iterator and only populate the result cache on demand. We actually populate the result cache 100 elements at a time, rather than one at a time for efficiency, but this is a real win when the resultset contains 10,000 objects for example.

This also provides an efficient boolean (nonzero) test that doesn't use up a lot of memory if you don't read all the results.

Refs #2430, #5987.

01/26/08 07:28:38 changed by mtredinnick

  • keywords changed from iterator __iter__ qs-rf to iterator __iter__ qs-rf-fixed.

04/26/08 21:50:16 changed by mtredinnick

  • status changed from new to closed.
  • resolution set to fixed.

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658


Add/Change #5987 (__iter__ on a QuerySet doesn't return an efficient iterator)




Change Properties
Action