Opened 3 years ago

Last modified 3 years ago

#33636 closed New feature

BulkProcessMixin on models.Model — at Version 5

Reported by: Myung Eui Yoon Owned by: Myung Eui Yoon
Component: Database layer (models, ORM) Version: 4.0
Severity: Normal Keywords: bulk, model
Cc: Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Myung Eui Yoon)

New Feature for convinient context manager for bulk create/update.
Just inherit MoModelBulkProcessMixin on Model class.

        class BulkyModel(models.Model, ModelBulkProcessMixin):
            id = models.BigAutoField(primary_key=True)
            name = models.CharField(max_length=10, null=False)

        names = [f"name-{num}" for num in range(100_100_000)]

        # Case : Raw bulk_create
        objs = [BulkyModel(name=name) for name in names]
        BulkyModel.objects.bulk_create(
            objs
        )  # We should maintain big array size and this leades to OOM error

        # Case : Chunked bulk_create
        objs = list()
        for name in names:
            obj = BulkyModel(name=name)
            objs.append(obj)
            if len(objs) > 10_1000:
                BulkyModel.objects.bulk_create(objs)
                objs.clear()
        if len(objs) > 0:
            BulkyModel.objects.bulk_create(objs)
            objs.clear()

        # Case : With ModelBulkProcessMixin
        with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
            for name in names:
                bulk.add(BulkyModel(name=name))

https://github.com/django/django/pull/15578

Change History (5)

comment:1 by Myung Eui Yoon, 3 years ago

Owner: changed from nobody to Myung Eui Yoon
Status: newassigned

comment:2 by Myung Eui Yoon, 3 years ago

Description: modified (diff)
Has patch: set

comment:3 by Myung Eui Yoon, 3 years ago

Description: modified (diff)

comment:4 by Myung Eui Yoon, 3 years ago

Description: modified (diff)

comment:5 by Myung Eui Yoon, 3 years ago

Description: modified (diff)
Note: See TracTickets for help on using tickets.
Back to Top