Changes between Version 2 and Version 3 of Ticket #33636


Ignore:
Timestamp:
Apr 11, 2022, 9:16:56 AM (3 years ago)
Author:
Myung Eui Yoon
Comment:

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #33636 – Description

    v2 v3  
     1
    12
    23{{{
    3         """
    44        class BulkyModel(models.Model, ModelBulkProcessMixin):
    55            id = models.BigAutoField(primary_key=True)
    66            name = models.CharField(max_length=10, null=False)
    7    
    8         With ModelBulkProcessMixin , We could minimize memory usage.
    9         Without ModelBulkProcessMixin, We shold maintain bulk array size up to 100_000
    10         or manually maintain arraysize up to batch_size like 10_000
    117
    12         if len(chunked_list)>10_000:
    13             Model.objects.bulk_create(chunked_list)
     8        names = [f"name-{num}" for num in range(100_100_000)]
    149
    15         and check remain in list again at the end.
    16        
    17         if len(chunked_list)>0:
    18             Model.objects.bulk_create(chunked_list)
    19         """
     10        # Case : Raw bulk_create
     11        objs = [BulkyModel(name=name) for name in names]
     12        BulkyModel.objects.bulk_create(
     13            objs
     14        )  # We should maintain big array size and this leades to OOM error
    2015
    21         names = [f"name-{num}" for num in range(100_000)]
     16        # Case : Chunked bulk_create
     17        objs = list()
     18        for name in names:
     19            obj = BulkyModel(name=name)
     20            objs.append(obj)
     21            if len(objs) > 10_1000:
     22                BulkyModel.objects.bulk_create(objs)
     23                objs.clear()
     24        if len(objs) > 0:
     25            BulkyModel.objects.bulk_create(objs)
     26            objs.clear()
    2227
     28        # Case : With ModelBulkProcessMixin
    2329        with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
    2430            for name in names:
    2531                bulk.add(BulkyModel(name=name))
    26 
    27         self.assertEqual(100_000, BulkyModel.objects.all().count())
    2832}}}
    2933
     34
    3035https://github.com/django/django/pull/15577
Back to Top