Opened 5 months ago

Closed 7 days ago

#29244 closed Cleanup/optimization (fixed)

Paginator.count() may incorrectly silence AttributeError and TypeError

Reported by: Rob Jauquet Owned by: nobody
Component: Core (Other) Version: 2.0
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Rob Jauquet)

With this code:

from import SearchQuery, Value, Func
from django.core.paginator import Paginator
from django.db.models import F, TextField, DateTimeField, Models
from django.utils import timezone

class Entry(Model):
    text = TextField()
    published_on = DateTimeField(

class Headline(Func):
    function = 'ts_headline'
    output_field = TextField()

    def __init__(self, text, query, options=None):
        extra = {}
        expressions = [text, query]
        if options:
            opt_str = ''
            for key, value in options.items():
                opt_str += f'{key}={value},'
        super().__init__(*expressions, **extra)

entries = Entry.objects.annotate(
        options={'HighlightAll': False},

paginator = Paginator(entries, 20)

# evaluates the queryset

Passing options by dict to the Func subclass causes the count step for pagination to fail silently in line 85 (1) which is in turn caused by a silenced TypeError during the hash in line 372 (2).

Then, because of the fallback to len on line 90 (3) the entire queryset is evaluated instead of evaluating just the specified page (and in this specific case, calling ts_headline on every row instead of just the first page).

The correct way of passing options using **options instead of options=None fixes this case, but the silent error and fallback to calling len on a queryset causes an unexpected expensive query when using pagination.


Change History (7)

comment:1 Changed 5 months ago by Tim Graham

Do you have a proposal of how to address this? By the way, if you press the "y" key when viewing a GitHub page, you can get a link that includes a commit hash. Without that, the links can quickly go stale as line numbers change.

comment:2 Changed 5 months ago by Rob Jauquet

Description: modified (diff)

comment:3 Changed 5 months ago by Rob Jauquet

Thanks for the tip. I've updated the links.

For my use case, it would make sense to not call len on a queryset in and instead surface the exception. The len fallback could be used only if the thing passed in is not a queryset.

comment:4 Changed 5 months ago by Tim Graham

Summary: Passing dict to a Func subclass fails silently on hash stepPaginator.count() may incorrectly silence AttributeError and TypeError
Triage Stage: UnreviewedAccepted
Type: BugCleanup/optimization

It looks like the best solution would be to remove the try/except in Paginator.count(). AttributeError can be removed with hasattr(self.object_list, 'count') and TypeError with some checking that count is a method without any arguments. Perhaps a new function in django/utils/ would help with that.

comment:5 Changed 8 days ago by Tim Graham

Has patch: set

comment:6 Changed 7 days ago by Tim Graham <timograham@…>

In f1bf069:

Refs #29244 -- Fixed django.utils.inspect.method_has_no_args() for bound methods.

comment:7 Changed 7 days ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In 3767c7f:

Fixed #29244 -- Prevented Paginator.count() from silencing TypeError and AttributeError.

Note: See TracTickets for help on using tickets.
Back to Top