Opened 3 weeks ago

Closed 2 weeks ago

#36606 closed Cleanup/optimization (fixed)

Optimize QuerySet.values_list(flat=True) with no fields

Reported by: Adam Johnson Owned by: Adam Johnson
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Adam Johnson)

Currently, QuerySet.values_list() ensures that no more than 1 field is set (source):

        if flat and len(fields) > 1:
            raise TypeError(
                "'flat' is not valid when values_list is called with more than one "
                "field."
            )

However, it also allows the case where *no* fields are declared, for which all fields are fetched, only to throw away all but the first one (source):

class FlatValuesListIterable(BaseIterable):
    """
    Iterable returned by QuerySet.values_list(flat=True) that yields single
    values.
    """

    def __iter__(self):
        queryset = self.queryset
        compiler = queryset.query.get_compiler(queryset.db)
        for row in compiler.results_iter(
            chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size
        ):
            yield row[0]

I think we can optimize this case to select only the first field in the model instead, maintaining semantics while avoiding overfetching.

This case also seems untested with the values_list() tests in tests/lookup/tests.py, so we'd want to add a test there.

Change History (8)

comment:1 by Adam Johnson, 3 weeks ago

Description: modified (diff)

comment:2 by Adam Johnson, 3 weeks ago

Owner: set to Adam Johnson
Status: newassigned

comment:3 by Simon Charette, 3 weeks ago

I wonder if we should take the opportunity to deprecate values_list(flat=True) instead by changing the check to be len(fields) != 1 as it seems like an omission in the API and should be a rare occurrence in code bases so I'm not convinced it's worth supporting it.

comment:4 by Adam Johnson, 3 weeks ago

Yes, I'm inclined to agree. It's not clear when reading what values_list(flat=True) would do.

I think we can make the optimization *and* deprecate, though.

comment:5 by Adam Johnson, 3 weeks ago

Has patch: set

comment:6 by Jacob Walls, 2 weeks ago

Triage Stage: UnreviewedAccepted

Agree we should deprecate, happy to take a follow-up PR as Refs ... this one.

comment:7 by Jacob Walls <jacobtylerwalls@…>, 2 weeks ago

In 2336d5d:

Refs #36606 -- Added tests for QuerySet.values_list(flat=True) without fields.

comment:8 by Jacob Walls <jacobtylerwalls@…>, 2 weeks ago

Resolution: fixed
Status: assignedclosed

In 2063c88:

Fixed #36606 -- Optimized QuerySet.values_list(flat=True) without fields.

Note: See TracTickets for help on using tickets.
Back to Top