Opened 13 years ago

Closed 7 years ago

#15648 closed New feature (fixed)

Allow QuerySet.values_list() to return a namedtuple

Reported by: Paul Miller Owned by: Sergey Fedoseev
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: namedtuple, tuple, queryset
Cc: kmike84@…, cg@…, dougal85@…, ShawnMilo, paulmillr@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Python 2.6 supports named tuples. Information about field names is stored in the tuple class, so there's no overhead like in dictionaries.
I propose to use them in querysets instead of values() / values_list().

qs = Items.objects.filter(...).namedtuples('title', 'amount', 'price')
for item in qs:
    print item.title, item.amount
    total += item.amount * item.price

Patch:

from itertools import imap
from collections import namedtuple

# python 2.5 doesn't support named tuples, so we can use this http://code.activestate.com/recipes/500261/

from django.db.models.query import ValuesQuerySet

class NamedTuplesQuerySet(ValuesQuerySet):
    def iterator(self):
        # get field names 
        extra_names = self.query.extra_select.keys()
        field_names = self.field_names
        aggregate_names = self.query.aggregate_select.keys()
        names = extra_names + field_names + aggregate_names
       
        # create named tuple class
        tuple_cls = namedtuple('%sTuple' % self.model.__name__, names)

        results_iter = self.query.get_compiler(self.db).results_iter()
        # wrap every string with our named tuple
        return imap(tuple_cls._make, results_iter)
from django.db.models.query import QuerySet

def namedtuples(self, *fields):
    return self._clone(klass=NamedTuplesQuerySet, setup=True, _fields=fields)
QuerySet.namedtuples = namedtuples

Attachments (2)

namedtuples.patch (11.2 KB ) - added by Paul Miller <paulmillr@…> 13 years ago.
Named tuples patch
values_list_namedtuples.diff (5.1 KB ) - added by Anssi Kääriäinen 12 years ago.

Download all attachments as: .zip

Change History (24)

comment:1 by Mikhail Korobov, 13 years ago

Cc: kmike84@… added

comment:2 by me@…, 13 years ago

Why not

Item.objects.all().only(*names_list)

?

in reply to:  2 comment:3 by Paul Miller, 13 years ago

Replying to me@…:

Why not

Item.objects.all().only(*names_list)

?

  1. repr(). values / values lists etc. are much easier to debug.
  2. deferred objects (only creates them) are not iterables

comment:4 by Adrian Holovaty, 13 years ago

Needs tests: set
Patch needs improvement: set
Triage Stage: UnreviewedAccepted

I like the idea of introducing a QuerySet that returns named tuples! Not crazy about the name, but I can't think of a better one at the moment.

Can you create an actual .patch file, include unit tests and include a fallback namedtuple implementation for our Python 2.4/2.5 users? A good place for that would be in django/utils/datastructures.py.

comment:5 by Luke Plant, 13 years ago

Type: New feature

comment:6 by Luke Plant, 13 years ago

Severity: Normal

comment:7 by Christopher Grebs, 13 years ago

Cc: cg@… added

comment:8 by Dougal Matthews, 13 years ago

Cc: dougal85@… added
Easy pickings: unset
Needs documentation: set

comment:9 by ShawnMilo, 13 years ago

Cc: ShawnMilo added

by Paul Miller <paulmillr@…>, 13 years ago

Attachment: namedtuples.patch added

Named tuples patch

comment:10 by Paul Miller <paulmillr@…>, 13 years ago

Cc: paulmillr@… added
Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
UI/UX: unset

I've added necessary tests and fallback implementation for Python 2.5 users.

comment:11 by Paul Miller, 13 years ago

Owner: changed from nobody to Paul Miller
Status: newassigned

comment:12 by Paul Miller, 13 years ago

Triage Stage: AcceptedReady for checkin

I don't know much about ticket triaging, but I think that «ready for checkin» would be more correct for this

comment:13 by Jannis Leidel, 13 years ago

Needs documentation: set
Patch needs improvement: set
Triage Stage: Ready for checkinAccepted

This definitely isn't ready for checkin yet as there are no docs.

comment:14 by Jacob, 12 years ago

milestone: 1.4

Milestone 1.4 deleted

comment:15 by Anssi Kääriäinen, 12 years ago

Why the namedtuples API? Is there some reason values_list should not return namedtuples directly? It should be backwards compatible, as index based fetching still work. I believe there will not be any serious performance impact.

If the performance is acceptable, then raw cursors should return named tuples, too. That would be a really nice addition for raw SQL users. But of course, this is another ticket's problem.

comment:16 by Anssi Kääriäinen, 12 years ago

I have a very quick implementation of returning namedtuples from .values_list(). The patch seems to be backwards compatible, that is, all tests pass.

The performance penalty seems to be about 20% for one-field values_list() call. It is a good question if that is acceptable. I guess yes as namedtuples are _much_ nicer than regular tuples. On the other hand the main use case for values_list is performance.

The results are using SQLite to fetch 200 objects with one field containing a four char value, and then additional fields containing NULLs. values_list with one field gives the 20% performance penalty, 5 fields gives 15% penalty. Notably in-memory SQLite with fields containing practically no data is pretty much the worst-case. I guess the real-world penalty will be in 5%-20% range depending on use case. My opinion is that getting namedtuples is worth it.

I guess similar performance penalty would be paid if cursors were to return named tuples directly, and that is even tougher call to make. Cursors are even more performance critical, and every query made by Django would need to pay the penalty. Although when creating model instance for example that penalty would be hidden by overhead of model.__init__.

It is of course possible to add the .namedtuples() API, but then we have three things doing nearly the same thing, and two doing exactly the same thing from user perspective.

by Anssi Kääriäinen, 12 years ago

comment:17 by anonymous, 11 years ago

This would be really, really nice.

comment:18 by suprzer0@…, 10 years ago

Perhaps adding a keyword argument to values_list() for namedtuples? something like

Item.objects.values_list(*names_list, named=True)

With named defaulting to False.

Calling .values_list(*names_list, named=True, flat=True) would need to throw some kind of exception I'd imagine.

comment:19 by Tim Graham, 9 years ago

Summary: [Feature request] NamedTupleQuerySetAllow QuerySet.values_list() to return a namedtuple

comment:20 by Andrii Soldatenko, 7 years ago

Does anyone works on this FR? I would like to help.
What's the final decision?

Do I need use this approach?
Item.objects.values_list(*names_list, named=True)

thanks.

comment:21 by Sergey Fedoseev, 7 years ago

Needs documentation: unset
Owner: changed from Paul Miller to Sergey Fedoseev
Patch needs improvement: unset

comment:22 by Tim Graham <timograham@…>, 7 years ago

Resolution: fixed
Status: assignedclosed

In f3c95621:

Fixed #15648 -- Allowed QuerySet.values_list() to return a namedtuple.

Note: See TracTickets for help on using tickets.
Back to Top