Opened 9 years ago
Last modified 8 years ago
#28231 closed Cleanup/optimization
bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided — at Version 2
| Reported by: | Nir Izraeli | Owned by: | nobody |
|---|---|---|---|
| Component: | Documentation | Version: | 1.11 |
| Severity: | Normal | Keywords: | |
| Cc: | nirizr@…, Botond Béres, Tim Martin | Triage Stage: | Ready for checkin |
| Has patch: | yes | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description (last modified by )
When batch_size provided in bulk_create, a user might assume (As I myself have) that objs iterable will not be iterated more than batch_size times at once, and that no more than roughly batch_size Model objects reside in memory at any given time.
When using bulk_create for relatively big sets of objects provided by a generator object, it would be prefered to avoid iterating over the entire generator object. Moreover, if not iterating over the generator object is deemed unnecessary or out-of-scope for django it would be prefered to make a comment on said behavior in documentation.
I suggest two possible solutions:
- Document this behavior (
bulk_createconverts passedobjsiterator to a list), or - Avoid doing so when
batch_sizeis given (or default is other thanNone. i.e. sqlite).
I did not research the possibility of avoiding the list conversion, but if that solution is accepted by the community I volunteer to investigate further and claim this ticket.
Change History (2)
comment:1 by , 9 years ago
| Cc: | added |
|---|
comment:2 by , 9 years ago
| Description: | modified (diff) |
|---|---|
| Summary: | bulk_create: avoid iterating `objs` more than necessary when `bulk_size` is provided → bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided |
CCing myself to receive updates.