Opened 8 years ago
Last modified 7 years ago
#28231 closed Cleanup/optimization
bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided — at Version 2
Reported by: | Nir Izraeli | Owned by: | nobody |
---|---|---|---|
Component: | Documentation | Version: | 1.11 |
Severity: | Normal | Keywords: | |
Cc: | nirizr@…, Botond Béres, Tim Martin | Triage Stage: | Ready for checkin |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
When batch_size
provided in bulk_create
, a user might assume (As I myself have) that objs
iterable will not be iterated more than batch_size
times at once, and that no more than roughly batch_size
Model objects reside in memory at any given time.
When using bulk_create
for relatively big sets of objects provided by a generator object, it would be prefered to avoid iterating over the entire generator object. Moreover, if not iterating over the generator object is deemed unnecessary or out-of-scope for django it would be prefered to make a comment on said behavior in documentation.
I suggest two possible solutions:
- Document this behavior (
bulk_create
converts passedobjs
iterator to a list), or - Avoid doing so when
batch_size
is given (or default is other thanNone
. i.e. sqlite).
I did not research the possibility of avoiding the list conversion, but if that solution is accepted by the community I volunteer to investigate further and claim this ticket.
Change History (2)
comment:1 by , 8 years ago
Cc: | added |
---|
comment:2 by , 8 years ago
Description: | modified (diff) |
---|---|
Summary: | bulk_create: avoid iterating `objs` more than necessary when `bulk_size` is provided → bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided |
CCing myself to receive updates.