Code

Opened 6 years ago

Closed 6 years ago

#5987 closed (fixed)

__iter__ on a QuerySet doesn't return an efficient iterator

Reported by: kenneth.arnold@… Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: iterator __iter__ qs-rf-fixed
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

For a loop such as:

for item in Model.objects.all():
  print item # or whatever

the first line eats up all system memory for sufficiently large Models.

Replacing it with Model.objects.all().iterator() improves the situation a lot. Why doesn't iter return iterator()?

(I see that iter can use a cache. Is there a reason not to compute the cache on-the-fly?)

Attachments (0)

Change History (4)

comment:1 Changed 6 years ago by mtredinnick

  • Keywords qs-rf added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

Since we cache the results anyway, this is all a bit of a non-event (because the memory will be used up anyway as you access the results) unless you're being very lazy with your queries and selecting way more information than you're using.

It's being changed in queryset-refactor in any case, for other reasons, so this ticket will be closed when that branch is merged.

comment:2 Changed 6 years ago by mtredinnick

(In [7030]) queryset-refactor: Converted the queryset iterator to be a real iterator and
only populate the result cache on demand. We actually populate the result cache
100 elements at a time, rather than one at a time for efficiency, but this is a
real win when the resultset contains 10,000 objects for example.

This also provides an efficient boolean (nonzero) test that doesn't use up
a lot of memory if you don't read all the results.

Refs #2430, #5987.

comment:3 Changed 6 years ago by mtredinnick

  • Keywords qs-rf-fixed added; qs-rf removed

comment:4 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.