Unnecessary deepcopying of QuerySet inside filter() method results in slower execution
|Reported by:||zimnyx||Owned by:||nobody|
|Component:||Database layer (models, ORM)||Version:||master|
|Has patch:||no||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
When building query using multiple filter() calls like that:
qs = MyModel.objects.filter(...).filter(...).filter(...)
it results in sql.Query.clone() for every filter() call.
Query.clone() uses deepcopy() for "where", "having", "aggregates" and "deferred_loading" properties:
obj.where = deepcopy(self.where, memo=memo) obj.having = deepcopy(self.having, memo=memo) obj.aggregates = deepcopy(self.aggregates, memo=memo) obj.deferred_loading = deepcopy(self.deferred_loading, memo=memo)
... which can be time-consumig operation.
My question is, do we really need deepcopy when we call filter() like this?
I mean we only need QuerySet created by last filter() call, so we can overwrite previous QuerySet instead of creating a (deep)copy.
Maybe adding flag for filter() method would be possible: filter(overwrite=False).
I attach profiler results for non-trivial query run on my Ubuntu box: CPU: Intel(R) Celeron(R) M CPU 530 @ 1.73GHz, Python 2.6.5 / MySQL 5.1@InnoDB
Results show that for complicated query, query building time can be far larger than executing SQL and filling model instances with results.
I tested two version of the query:
# multi-filter result = Zero.objects.select_related('one__two__three') \ .filter(field_x=None) \ .filter(field_t='P') \ .filter(one__id__gt=2) \ .filter(one__field_j__isnull=True) \ .filter(one__two__id__isnull=False) \ .filter(one__two__field_y__year=2009) \ .filter(one__two__three__field_j__isnull=True) \ .filter(one__two__three__field_z__gt=2) \ [:1] len(result)
# filter-once result = Zero.objects.select_related('one__two__three') \ .filter(field_x=None, field_t='P', one__id__gt=2, one__field_j__isnull=True, one__two__id__isnull=False, one__two__field_y__year=2009, one__two__three__field_j__isnull=True, one__two__three__field_z__gt=2 )[:1] len(result)
Change History (4)
Changed 5 years ago by zimnyx
comment:1 Changed 4 years ago by russellm
- Needs documentation unset
- Needs tests unset
- Patch needs improvement unset
- Resolution set to invalid
- Status changed from new to closed
comment:2 Changed 4 years ago by zimnyx
- Resolution invalid deleted
- Status changed from closed to reopened