Opened 3 years ago

Closed 3 years ago

#32663 closed Cleanup/optimization (wontfix)

Remove Error raising on annotation & distinct call

Reported by: Yovel Cohen Owned by: nobody
Component: Database layer (models, ORM) Version: 3.2
Severity: Normal Keywords: SQLCompiler ORM Query
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Yovel Cohen)

if you try to call distinct on an annotated query, it sometimes works, sometimes get's ignore, and sometimes raises an error.
I had a query facing all three scenarios and solving the last one was the one that worked out.

the error being raised is:

NotImplementedError: annotate() + distinct(fields) is not implemented.

after looking at the source code for the exception at django.db.models.sql.compiler.SQLCompiler
row 594

if grouping:
        if distinct_fields:
           raise NotImplementedError('annotate() + distinct(fields) is not implemented.')
        order_by = order_by or self.connection.ops.force_no_ordering()
        result.append('GROUP BY %s' % ', '.join(grouping))
# rest of the as_sql() method

after just removing the distinct_fields condition:

if grouping:
        order_by = order_by or self.connection.ops.force_no_ordering()
        result.append('GROUP BY %s' % ', '.join(grouping))
# rest of the as_sql() method

it works, at least in the following ways I tried (annotations are just made up for sake of the example)

model_scores_latest_date_annotation = Max('model_scores__date')
latest_score_annotation = Case(When(model_scores__date=F('latest_date'), then='model_scores__score')
base_query_set = base_query_set = (Model.objects.
                  filter(**filters).
                  alias(latest_date=model_scores_latest_date_annotation).
                  values(ID).
                  annotate(latest_score=latest_score_annotation, latest_date=model_scores_latest_date_annotation)
                  )

all the following distinct calls worked:

query_set = base_query_set.order_by('latest_date').distinct('id', 'latest_date')

query_set = base_query_set.distinct('id')

query_set = base_query_set.distinct('id', 'latest_date')

which makes me think that the as_sql method on SQLCompiler can handle more cases easily and this just fell beneath the cracks.

Change History (5)

comment:1 by Yovel Cohen, 3 years ago

Description: modified (diff)

comment:2 by Yovel Cohen, 3 years ago

Description: modified (diff)

comment:3 by Yovel Cohen, 3 years ago

Description: modified (diff)

comment:4 by Simon Charette, 3 years ago

There's a valid reason why this error is raised as explained in these comment https://code.djangoproject.com/ticket/6422#comment:59; it's less about the validity of the generated SQL and more about the correctness of the returned results.

Given the rare cases where mixing aggregation and distinct is necessary and the complexity involved in getting it right I'm of opinion that we should keep disallowing it.

comment:5 by Mariusz Felisiak, 3 years ago

Resolution: wontfix
Status: newclosed

I agree with Simon.

Note: See TracTickets for help on using tickets.
Back to Top