Opened 7 years ago

Last modified 6 years ago

#28231 closed Cleanup/optimization

bulk_create: avoid iterating `objs` more than necessary when `bulk_size` is provided — at Initial Version

Reported by: Nir Izraeli Owned by: nobody
Component: Documentation Version: 1.11
Severity: Normal Keywords:
Cc: nirizr@…, Botond Béres, Tim Martin Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When bulk_size provided in bulk_create, a user might assume (As I myself have) that objs iterable will not be iterated more than bulk_size times at once, and that no more than roughly bulk_size Model objects reside in memory at any given time.

When using bulk_create for relatively big sets of objects provided by a generator object, it would be prefered to avoid iterating over the entire generator object. Moreover, if not iterating over the generator object is deemed unnecessary or out-of-scope for django it would be prefered to make a comment on said behavior in documentation.

I suggest two possible solutions:

  1. Document this behavior (bulk_create converts passed objs iterator to a list), or
  2. Avoid doing so when bulk_size is given (or default is other than None. i.e. sqlite).

I did not research the possibility of avoiding the list conversion, but if that solution is accepted by the community I volunteer to investigate further and claim this ticket.

Change History (0)

Note: See TracTickets for help on using tickets.
Back to Top