Opened 16 months ago

Last modified 9 months ago

#22503 new New feature

Inconsistent behavior when a QuerySet is sliced

Reported by: ajaest Owned by: nobody
Component: Database layer (models, ORM) Version: 1.6
Severity: Normal Keywords: QuerySet, filter, slice
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I think that the current behavior of the QuerySet.slice method is not consistent. Just an example:

def limited_filtering(max_results=10, *args, **kwargs){
    """
    Returns a Queryset
    """

    returns SomeModel.filter(*args, **kwargs)[:max_results]
}

# ...

def other_method_somewhere(queryset){
    """
    Expects a Queryset of SomeModel and returns the same Queryset filtered again
    """

    return queryset.filter(some_other_criteria)
}

other_method_somewhere(limited_filtering(*args, **kwargs)); # raises "AssertionError: Cannot filter a query once a slice has been taken."

Given the previous code, it is legit and fair that other_method_somewhere expects, as in most situations, a filtrable QuerySet. Not allowing a sliced QuerySet to be filtered after being sliced adds an edge case difficult to handle when methods using a QuerySet type are somehow semantically far away from each other (and thus, they don't know if the other is returning an sliced QuerySet).

Without knowing the exact reasons why this was implemented this way, the possible solutions I can come with are:

  • The most natural one, to make the sliced QuerySets filtrable.
  • To make all the returned query sets in which can_filter == False to return a new type of QuerySet, something like InmutableQuerySet in order to take advantage of other methods related to them (values_list, etc.).
  • To make QuerySet's slices return list or other type of collection (I guess this is impossible without breaking backwards compatibility).

Actually, the last case is supposedly the way one would expect it to work according to the documentation:

https://docs.djangoproject.com/en/dev/ref/models/querysets/#when-querysets-are-evaluated

Slicing. As explained in Limiting QuerySets, a QuerySet can be sliced, using Python’s array-slicing syntax. Slicing an unevaluated QuerySet usually returns another unevaluated QuerySet, but Django will execute the database query if you use the “step” parameter of slice syntax, and will return a list. Slicing a QuerySet that has been evaluated (partially or fully) also returns a list.

By the way, there is also no reference in the documentation(or at least I could not find it) to the public method QuerySet.can_filter.

Change History (7)

comment:1 Changed 15 months ago by timo

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Type changed from Bug to New feature

There was a recent django-dev thread about this. I have not taken time to fully read it and determine if any consensus was reached.

comment:2 Changed 15 months ago by timo

  • Triage Stage changed from Unreviewed to Accepted

If we don't do this, let's document why not.

comment:3 Changed 14 months ago by jorgecarleitao

I will map the discussion in the thread to here: I believe the consensus was that this cannot be done:

Assume other_method_somewhere uses a filter. The sequence would be

queryset.filter(first_clause)[:10].filter(second_clause)

Most people would assume (as pointed out in the thread) that the second clause would be applied to the last 10 entries, when it is not. Doing that is not possible in SQL without
subqueries and thus we explicitly forbid it.

comment:4 Changed 9 months ago by tchaumeny

See also https://groups.google.com/forum/#!topic/django-developers/ELqU2xt_Qo0 (sliced querysets not usable in a ModelChoiceField because not filterable).

comment:5 Changed 9 months ago by Tim Graham <timograham@…>

In 327df551e89a505c5756becee97c40198f38aff2:

Fixed #23817 -- Updated docs on QuerySet evaluation

Removed inaccurate info about partial evaluation after refs #18702.
Added information on modifying sliced QuerySets; refs #22503.

comment:6 Changed 9 months ago by Tim Graham <timograham@…>

In 614dd44d0d2e5f8f12256835c3453f420f54c3b4:

[1.6.x] Fixed #23817 -- Updated docs on QuerySet evaluation

Removed inaccurate info about partial evaluation after refs #18702.
Added information on modifying sliced QuerySets; refs #22503.

Backport of 327df551e89a505c5756becee97c40198f38aff2 from master

comment:7 Changed 9 months ago by Tim Graham <timograham@…>

In 593353d8af5b4b8d1a1e627712fe68ed593961d0:

[1.7.x] Fixed #23817 -- Updated docs on QuerySet evaluation

Removed inaccurate info about partial evaluation after refs #18702.
Added information on modifying sliced QuerySets; refs #22503.

Backport of 327df551e89a505c5756becee97c40198f38aff2 from master

Note: See TracTickets for help on using tickets.
Back to Top