Opened 10 years ago

Last modified 6 years ago

#22503 new New feature

Inconsistent behavior when a QuerySet is sliced

Reported by: Luis A. Arce Owned by: nobody
Component: Database layer (models, ORM) Version: 1.6
Severity: Normal Keywords: QuerySet, filter, slice
Cc: Дилян Палаузов Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I think that the current behavior of the QuerySet.slice method is not consistent. Just an example:

def limited_filtering(max_results=10, *args, **kwargs){
    """
    Returns a Queryset
    """

    returns SomeModel.filter(*args, **kwargs)[:max_results]
}

# ...

def other_method_somewhere(queryset){
    """
    Expects a Queryset of SomeModel and returns the same Queryset filtered again
    """

    return queryset.filter(some_other_criteria)
}

other_method_somewhere(limited_filtering(*args, **kwargs)); # raises "AssertionError: Cannot filter a query once a slice has been taken."

Given the previous code, it is legit and fair that other_method_somewhere expects, as in most situations, a filtrable QuerySet. Not allowing a sliced QuerySet to be filtered after being sliced adds an edge case difficult to handle when methods using a QuerySet type are somehow semantically far away from each other (and thus, they don't know if the other is returning an sliced QuerySet).

Without knowing the exact reasons why this was implemented this way, the possible solutions I can come with are:

  • The most natural one, to make the sliced QuerySets filtrable.
  • To make all the returned query sets in which can_filter == False to return a new type of QuerySet, something like InmutableQuerySet in order to take advantage of other methods related to them (values_list, etc.).
  • To make QuerySet's slices return list or other type of collection (I guess this is impossible without breaking backwards compatibility).

Actually, the last case is supposedly the way one would expect it to work according to the documentation:

https://docs.djangoproject.com/en/dev/ref/models/querysets/#when-querysets-are-evaluated

Slicing. As explained in Limiting QuerySets, a QuerySet can be sliced, using Python’s array-slicing syntax. Slicing an unevaluated QuerySet usually returns another unevaluated QuerySet, but Django will execute the database query if you use the “step” parameter of slice syntax, and will return a list. Slicing a QuerySet that has been evaluated (partially or fully) also returns a list.

By the way, there is also no reference in the documentation(or at least I could not find it) to the public method QuerySet.can_filter.

Change History (9)

comment:1 by Tim Graham, 10 years ago

Type: BugNew feature

There was a recent django-dev thread about this. I have not taken time to fully read it and determine if any consensus was reached.

comment:2 by Tim Graham, 10 years ago

Triage Stage: UnreviewedAccepted

If we don't do this, let's document why not.

comment:3 by jorgecarleitao, 10 years ago

I will map the discussion in the thread to here: I believe the consensus was that this cannot be done:

Assume other_method_somewhere uses a filter. The sequence would be

queryset.filter(first_clause)[:10].filter(second_clause)

Most people would assume (as pointed out in the thread) that the second clause would be applied to the last 10 entries, when it is not. Doing that is not possible in SQL without
subqueries and thus we explicitly forbid it.

comment:4 by Thomas C, 9 years ago

See also https://groups.google.com/forum/#!topic/django-developers/ELqU2xt_Qo0 (sliced querysets not usable in a ModelChoiceField because not filterable).

comment:5 by Tim Graham <timograham@…>, 9 years ago

In 327df551e89a505c5756becee97c40198f38aff2:

Fixed #23817 -- Updated docs on QuerySet evaluation

Removed inaccurate info about partial evaluation after refs #18702.
Added information on modifying sliced QuerySets; refs #22503.

comment:6 by Tim Graham <timograham@…>, 9 years ago

In 614dd44d0d2e5f8f12256835c3453f420f54c3b4:

[1.6.x] Fixed #23817 -- Updated docs on QuerySet evaluation

Removed inaccurate info about partial evaluation after refs #18702.
Added information on modifying sliced QuerySets; refs #22503.

Backport of 327df551e89a505c5756becee97c40198f38aff2 from master

comment:7 by Tim Graham <timograham@…>, 9 years ago

In 593353d8af5b4b8d1a1e627712fe68ed593961d0:

[1.7.x] Fixed #23817 -- Updated docs on QuerySet evaluation

Removed inaccurate info about partial evaluation after refs #18702.
Added information on modifying sliced QuerySets; refs #22503.

Backport of 327df551e89a505c5756becee97c40198f38aff2 from master

comment:8 by Дилян Палаузов, 6 years ago

Is this fixed? The last comments suggest so, but the status is "new".

comment:9 by Дилян Палаузов, 6 years ago

Cc: Дилян Палаузов added
Note: See TracTickets for help on using tickets.
Back to Top