Opened 81 minutes ago

Last modified 45 minutes ago

#37473 new Bug

count() and aggregate() build an empty SELECT when values() selects only aggregate annotations

Reported by: cheek-walnut Owned by:
Component: Database layer (models, ORM) Version: 5.2
Severity: Low Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by cheek-walnut)

When values() selects only an aggregate annotation, a following count(), or aggregate() over another field, generates a subquery with an empty SELECT list.

class Author(models.Model):
    name = models.CharField(max_length=20)
    age = models.IntegerField()

class Book(models.Model):
    title = models.CharField(max_length=20)
    pages = models.IntegerField()
    author = models.ForeignKey(Author, models.CASCADE, related_name="books")

With two authors (ages 40 and 30) and one book each:

>>> Author.objects.annotate(c=Count("books")).values("c").aggregate(m=Max("age"))
Traceback (most recent call last):
  ...
  File "django/db/backends/sqlite3/base.py", line 365, in execute
    return super().execute(query, params)
django.db.utils.OperationalError: near "FROM": syntax error

Generated SQL (SQLite):

SELECT MAX("__col1") FROM (SELECT  FROM "app_author" LEFT OUTER JOIN "app_book" ON ("app_author"."id" = "app_book"."author_id") GROUP BY "app_author"."id", "app_author"."name", "app_author"."age") subquery

Expected {'m': 40}, as on Django 5.1, where the inner query selects "app_author"."age" AS "__col1". values_list("c") fails the same way; values("name", "c") and aggregate(m=Max("c")) work.

Regression in 65ad4ade74dc9208b9d686a451cd6045df0c9c3a (Refs #28900): the query works on its parent and fails on it. Reproduced on 5.2.19, 6.0.10, 6.1.3 and main with SQLite; 5.1.15 is not affected. On PostgreSQL (checked on 16), which accepts an empty SELECT list, count() returns the right number but aggregate(m=Max("age")) fails with column "__col1" does not exist.

The cause appears to be if self.selected: in Query.set_annotation_mask() (django/db/models/sql/query.py:2568 at 5bd95c57a2) and Query.add_annotation() (:1248), which treats an empty selected dict like None: after get_aggregation() masks the unused c annotation, selected is {} and the __col1 column added for the aggregate is never added to it. count() also goes through the fallback that selects the primary key when nothing else is selected (query.py:595-605), which only sets inner_query.select and is ignored once selected is {}, so changing the two checks to is not None fixes aggregate() but not count().

AI disclosure: found during a code audit with Claude Code 2.1.317 (Claude Opus 5.5), which also reduced the test case and ran it on the versions above and on 65ad4ade74 and its parent. The PostgreSQL behavior and the count() note came from an independent review with ChatGPT Pro and were re-run with Claude Code. I reviewed this report and re-ran the reproduction on main and stable/5.1.x.

Change History (3)

comment:1 by cheek-walnut, 64 minutes ago

Summary: QuerySet.aggregate() crashes when values() selects only an aggregate annotation → QuerySet.count() and aggregate() crash when values() selects only aggregate annotations

On SQLite, the same crash is easier to hit through count(), e.g. with Paginator:

>>> qs = Author.objects.annotate(n=Count("books")).values_list("n", flat=True)
>>> qs.count()
django.db.utils.OperationalError: near "FROM": syntax error
>>> Paginator(qs.order_by("n"), 10).count
django.db.utils.OperationalError: near "FROM": syntax error

The generated SQL is again SELECT COUNT(*) FROM (SELECT FROM "app_author" ... GROUP BY ...) subquery; on 5.1.15 the inner query selects "app_author"."id" AS "col1". aggregate() also fails with a filter on the annotation (.filter(n__gt=0).values("n").aggregate(m=Max("age"))) and with aggregate(avg=Avg("age")).

All of these work on 5.1.15 and on the parent of 65ad4ade74, and fail on 65ad4ade74, 5.2.19, 6.0.10, 6.1.3 and main.

AI disclosure: these variants were also found and run with Claude Code 2.1.317 (Claude Opus 5.5); I re-ran them on main and stable/5.1.x.

Last edited 45 minutes ago by cheek-walnut (previous) (diff)

comment:2 by cheek-walnut, 59 minutes ago

Description: modified (diff)

comment:3 by cheek-walnut, 45 minutes ago

Description: modified (diff)
Summary: QuerySet.count() and aggregate() crash when values() selects only aggregate annotations → count() and aggregate() build an empty SELECT when values() selects only aggregate annotations
Note: See TracTickets for help on using tickets.
Back to Top