Opened 8 years ago

Last modified 7 years ago

#28231 closed Cleanup/optimization

bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided — at Version 2

Reported by: Nir Izraeli Owned by: nobody
Component: Documentation Version: 1.11
Severity: Normal Keywords:
Cc: nirizr@…, Botond Béres, Tim Martin Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Nir Izraeli)

When batch_size provided in bulk_create, a user might assume (As I myself have) that objs iterable will not be iterated more than batch_size times at once, and that no more than roughly batch_size Model objects reside in memory at any given time.

When using bulk_create for relatively big sets of objects provided by a generator object, it would be prefered to avoid iterating over the entire generator object. Moreover, if not iterating over the generator object is deemed unnecessary or out-of-scope for django it would be prefered to make a comment on said behavior in documentation.

I suggest two possible solutions:

  1. Document this behavior (bulk_create converts passed objs iterator to a list), or
  2. Avoid doing so when batch_size is given (or default is other than None. i.e. sqlite).

I did not research the possibility of avoiding the list conversion, but if that solution is accepted by the community I volunteer to investigate further and claim this ticket.

Change History (2)

comment:1 by Nir Izraeli, 8 years ago

Cc: nirizr@… added

CCing myself to receive updates.

comment:2 by Nir Izraeli, 8 years ago

Description: modified (diff)
Summary: bulk_create: avoid iterating `objs` more than necessary when `bulk_size` is providedbulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided
Note: See TracTickets for help on using tickets.
Back to Top