Opened 8 years ago
Last modified 8 years ago
#28231 closed Cleanup/optimization
bulk_create: avoid iterating `objs` more than necessary when `bulk_size` is provided — at Initial Version
| Reported by: | Nir Izraeli | Owned by: | nobody |
|---|---|---|---|
| Component: | Documentation | Version: | 1.11 |
| Severity: | Normal | Keywords: | |
| Cc: | nirizr@…, Botond Béres, Tim Martin | Triage Stage: | Ready for checkin |
| Has patch: | yes | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
When bulk_size provided in bulk_create, a user might assume (As I myself have) that objs iterable will not be iterated more than bulk_size times at once, and that no more than roughly bulk_size Model objects reside in memory at any given time.
When using bulk_create for relatively big sets of objects provided by a generator object, it would be prefered to avoid iterating over the entire generator object. Moreover, if not iterating over the generator object is deemed unnecessary or out-of-scope for django it would be prefered to make a comment on said behavior in documentation.
I suggest two possible solutions:
- Document this behavior (
bulk_createconverts passedobjsiterator to a list), or - Avoid doing so when
bulk_sizeis given (or default is other thanNone. i.e. sqlite).
I did not research the possibility of avoiding the list conversion, but if that solution is accepted by the community I volunteer to investigate further and claim this ticket.