Opened 6 years ago

Last modified 5 years ago

#22158 new New feature

Allow model level custom lookups

Reported by: Anssi Kääriäinen Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords:
Cc: Simon Charette Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


The idea is to allow DRY of common queries. For example, lets consider case where we have employee model with start and end dates. We want to filter all active employees. The basic query looks something like this:

Employee.objects.filter(start__lte=today(), Q(end__gte=today()) | (end__isnull=True))

it is possible to implement this as manager (say .active()) method, but that doesn't allow usage of the definition in related queries (that is, it is not possible to do Task.objects.filter(employee__active=True). Being able to do that would be a big win in DRYing up query generation.

The idea is to allow for class based lookups. I see two ways forward:

  1. Allow returning of Q() objects from Employee.get_model_lookup(lookup_name, value). Employee.get_model_lookup('active', True) would return Q(start__lte=today(), Q(end__gte=today()) | (end__isnull=True)).
  2. Allow returning SQL expressions from get_model_lookup(). The example case would be handled by directly constructing the needed SQL for the active query.

For related querying (that is, employee__active=True) the Q() object returned from get_model_lookup() must be rewritten to Q(employee__start__lte=today(), Q(employee__end__gte=today()) | (employee__end__isnull=True)). This should be doable inside the ORM.

For the example case the Q() way would be a lot easier to use. But there likely exists cases where one wants to use SQL directly, so allowing for returning Q-objects or SQL expressions seems like a good goal.

It might be handy to allow for registration of model lookups. This could allow for fields to register their own model level lookups, or for 3rd party apps to register lookups for all models.

Change History (4)

comment:1 Changed 6 years ago by Marc Tamlyn

Triage Stage: UnreviewedAccepted

It should definitely be able to handle Q objects as well as SQL as this is the 90% use case I think. Really nice feature idea though!

comment:2 Changed 6 years ago by ris

This would be incredibly helpful - I'm currently having to bodge together my own solution for making custom lookups that can refer to >1 field.

comment:3 Changed 5 years ago by Anssi Kääriäinen

I came up with another instance where this feature would be useful for one of my projects, and decided it might be time to move forward with this. The syntax I came up with was:

class User(models.Model):
    valid_from = models.DateField()
    valid_until = models.DateField(null=True, blank=True)

    def is_active_at(self, dt):
        return (self.valid_from < dt and
                   (self.valid_until is None or self.valid_until > dt))

    def is_active_at(cls, dt):
        return (Q(valid_from__lt=dt) &
                   (Q(valid_until__gt=dt) | Q(valid_until__isnull=True)))

    def valid_from_year(self):
        return self.valid_from.year

    def valid_from_year(cls):
        return Extract(F('valid_from'), 'year') 

So, the idea is that you could do both:




Similarly you could do both:



user.valid_from_year <= 2000

The idea would be that we use some decorators to mark methods for database lookups and expressions. When the transform/expression is referenced by the ORM, we first call the decorated method, then we resolve the expression. It might be worth it to pass the query to the methods, too.

Does this seem like an OK API?

comment:4 Changed 5 years ago by Simon Charette

Cc: Simon Charette added

I like your proposed API, it handles all the cases I can think of.

It also solves a limitation of custom managers used to filter their initial queryset, they can't be combined.

class Person(models.Model):
        (ROLE_AUTHOR, _('Author')),
        (ROLE_EDITOR, _('Editor')),
    dob = models.DateField()
    role = models.CharField(max_length=1, choices=ROLE_CHOICES)

    def author(cls):
        return Q(role=self.ROLE_AUTHOR)

    def born_90s(cls):
        return Q(dob__year__gte=1990, dob__year__lt=2000)

Person.objects.filter(author=True, born_90s=True)

Would you mind exposing this approach to the @developpers mailing list? I think we could get interesting feedback there.

Note: See TracTickets for help on using tickets.
Back to Top