id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
14474	Unnecessary deepcopying of QuerySet inside filter() method results in slower execution	Piotr Czachur	nobody	"When building query using multiple filter() calls like that:
{{{
    qs = MyModel.objects.filter(...).filter(...).filter(...)
}}}
it results in sql.Query.clone() for every filter() call.
Query.clone() uses deepcopy() for ""where"", ""having"", ""aggregates"" and ""deferred_loading"" properties:
{{{ 
        obj.where = deepcopy(self.where, memo=memo)
        obj.having = deepcopy(self.having, memo=memo)
        obj.aggregates = deepcopy(self.aggregates, memo=memo)
        obj.deferred_loading = deepcopy(self.deferred_loading, memo=memo)

}}}
... which can be time-consumig operation.

My question is, do we really need deepcopy when we call filter() like this?
I mean we only need QuerySet created by last filter() call, so we can overwrite previous QuerySet instead of creating a (deep)copy.
Maybe adding flag for filter() method would be possible:  filter(overwrite=False).


P.S.

I attach profiler results for non-trivial query run on my Ubuntu box: CPU: Intel(R) Celeron(R) M CPU 530 @ 1.73GHz, Python 2.6.5 / MySQL 5.1@InnoDB
Results show that for complicated query, query building time can be far larger than executing SQL and filling model instances with results.
I tested two version of the query:
{{{
    # multi-filter
    result = Zero.objects.select_related('one__two__three') \
        .filter(field_x=None) \
        .filter(field_t='P') \
        .filter(one__id__gt=2) \
        .filter(one__field_j__isnull=True) \
        .filter(one__two__id__isnull=False) \
        .filter(one__two__field_y__year=2009) \
        .filter(one__two__three__field_j__isnull=True) \
        .filter(one__two__three__field_z__gt=2) \
        [:1]
    len(result)
}}}
 and    
{{{
    # filter-once
    result = Zero.objects.select_related('one__two__three') \
        .filter(field_x=None, 
        field_t='P', 
        one__id__gt=2, 
        one__field_j__isnull=True, 
        one__two__id__isnull=False, 
        one__two__field_y__year=2009, 
        one__two__three__field_j__isnull=True, 
        one__two__three__field_z__gt=2 
        )[:1]
    len(result)
}}}

"		closed	Database layer (models, ORM)	dev		wontfix	deepcopy, filter		Unreviewed	0	0	0	0	0	0