Performance issue in date-based generic views

[Aymeric Augustin]

When BaseDateListView.allow_empty = False -- which is the default -- BaseDateListView.get_dated_queryset evaluates the unpaginated queryset, unnecessarily loading the entire table into memory.

Suggested fix:

Index: django/views/generic/dates.py
===================================================================
--- django/views/generic/dates.py	(revision 17859)
+++ django/views/generic/dates.py	(working copy)
@@ -194,7 +194,7 @@
         if not allow_future:
             qs = qs.filter(**{'%s__lte' % date_field: timezone.now()})
 
-        if not allow_empty and not qs:
+        if not allow_empty and not qs.exists():
             raise Http404(_(u"No %(verbose_name_plural)s available") % {
                     'verbose_name_plural': force_unicode(qs.model._meta.verbose_name_plural)
             })

At this point, I have no idea to capture this behavior in tests :(

[Aymeric Augustin]

Note: to work around this problem, you can set allow_empty = True.

[Claude Paroz]

I remember having discussed this on another ticket, but cannot find it again.

If you have a big queryset, your proposal makes sense, but the problem is that you are causing an extra query for all smaller querysets. As we cannot know the size of the queryset beforehand, it's a question about which scenario we want to privilege here.

[Aymeric Augustin]

Indeed, I vaguely remember a discussion about this too, but I can't find a ticket. Maybe it was on IRC. #16394 is related but doesn't discuss this specific problem.

To be precise, the problem only occurs when pagination is enabled. One would expect pagination to improve performance, but since allow_empty = False by default, it doesn't change anything. If you have a large prefetch_related (which is how I noticed the problem), performance becomes ugly quickly.

I'm attaching a new patch showing another technique: when pagination is enabled, use qs.exists(); when it isn't, use qs, since it will be loaded anyway.

[Claude Paroz]

I think it's a good compromise.

[Aymeric Augustin]

It's possible to write tests for this behavior with assertNumQueries.

[Aymeric Augustin]

In [17893]:

Fixed #18087 -- Prevented date-based generic views from loading entire tables in memory when pagination is enabled.