Annotating an aggregate function over a group including annotations or transforms followed by a column references crashes with IndexError

[Patrick Altman]

When I upgraded to 5.2 today, a number of my query sets broke tracebacks on building the query. I didn't notice anything in the BIC of the release notes.

I tossed together a project to make it easier to clone and repro:
​https://github.com/paltman/groupby-demo-dj52

But in summary, when I have some models like:

from django.db import models


class Bag(models.Model):
    kind = models.PositiveBigIntegerField(unique=True)


class Order(models.Model):
    blended_at = models.DateTimeField(null=True, default=None)
    gross_weight = models.FloatField(null=True, default=None)
    tare_weight = models.FloatField(null=True, default=None)
    total_tons = models.FloatField(null=True, default=None)
    blend_status = models.CharField(max_length=2, blank=True)
    bag_type = models.ForeignKey(Bag, on_delete=models.CASCADE, null=True)

and try to build a queryset like:

from django.db.models import Case, When, F, FloatField, Value, Sum

from .models import Order


TONS_ANNOTATION = dict(
    blended_tons=Case(
        When(
            gross_weight__gt=0,
            then=(F("gross_weight") - F("tare_weight")) / Value(2000)
        ),
        default=F("total_tons"),
        output_field=FloatField()
    )
)

dry_tons = Order.objects.filter(
    blended_at__date__range=["2025-03-01", "2025-03-31"],
).annotate(
    **TONS_ANNOTATION
).values(
    "blend_status",
    "blended_at__date",
    "bag_type__kind",
).annotate(
    tons=Sum("blended_tons")
)

I get the following traceback when trying to just print the query out:

>>> print(dry_tons.query)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.13/site-packages/django/db/models/sql/query.py", line 342, in __str__
    sql, params = self.sql_with_params()
                  ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/django/db/models/sql/query.py", line 350, in sql_with_params
    return self.get_compiler(DEFAULT_DB_ALIAS).as_sql()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 765, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup(
                                       ~~~~~~~~~~~~~~~~~~^
        with_col_aliases=with_col_aliases or bool(combinator),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 85, in pre_sql_setup
    self.setup_query(with_col_aliases=with_col_aliases)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 74, in setup_query
    self.select, self.klass_info, self.annotation_col_map = self.get_select(
                                                            ~~~~~~~~~~~~~~~^
        with_col_aliases=with_col_aliases,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 286, in get_select
    expression = cols[expression]
                 ~~~~^^^^^^^^^^^^
IndexError: tuple index out of range

Posted here earlier: ​https://forum.djangoproject.com/t/my-group-by-aggregations-are-breaking-in-5-2/40162/1 but after I posted I realized this seems like a bug just in the fact that it's been working for a while now in previous versions.

[Simon Charette]

I haven't figured out a solution yet but this is a regression due to #28900 (65ad4ade74dc9208b9d686a451cd6045df0c9c3a).

[Simon Charette]

​PR

[Mariusz Felisiak <felisiak.mariusz@…>]

In 543e17c:

Fixed #36292 -- Fixed crash when aggregating over a group mixing transforms and references.

Regression in 65ad4ade74dc9208b9d686a451cd6045df0c9c3a.

Refs #28900

Thanks Patrick Altman for the report.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 3176904:

[5.2.x] Fixed #36292 -- Fixed crash when aggregating over a group mixing transforms and references.

Regression in 65ad4ade74dc9208b9d686a451cd6045df0c9c3a.

Refs #28900

Thanks Patrick Altman for the report.

Backport of 543e17c4405dfdac4f18759fc78b190406d14239 from main