Aggregating when grouping on an ExpressionWrapper omits the expression from the group by

[Gordon Wrigley]

I ran into this with Postgres on Django 3.1.3, I'm not sure what other versions it exists on.

print(
    Fred.objects.annotate(
        bob_id__is_null=ExpressionWrapper(
            Q(bob_id=None), 
            output_field=BooleanField()
        )
    ).values(
        "bob_id__is_null"
    ).annotate(
        id__count=Count("id", distinct=True)
    ).values(
        "bob_id__is_null", 
        "id__count"
    ).query
)

SELECT 
    "main_fred"."bob_id" IS NULL AS "bob_id__is_null", 
    COUNT(DISTINCT "main_fred"."id") AS "id__count" 
FROM "main_fred"
GROUP BY "main_fred"."bob_id"

On the last line there the group by has dropped the "IS NULL"

[Gordon Wrigley]

For anyone else who ends up here you can Subquery your way out of this, although I've no idea what the performance implications of that are.

[Gordon Wrigley]

I did some further digging and this only occurs on 3.1, it works fine on 2.0, 2.1, 2.2 and 3.0

[Gordon Wrigley]

I think this is related to https://code.djangoproject.com/ticket/32007

[Gordon Wrigley]

To test this, given the implied model above, you can create 3 Fred objects, 2 with one value for bob_id and the third with a different value.
When you do the select on that you should see [{"bob_id__is_null": False, "id_count": 3}]
But instead you will get [{"bob_id__is_null": False, "id_count": 1}, {"bob_id__is_null": False, "id_count": 2}]

[Mariusz Felisiak]

Regression in df32fd42b84cc6dbba173201f244491b0d154a63 (backported in fdd2b01e8e12857aad2219a46a41bd9051ec8f8d).
Reproduced at 4cce1d13cfe9d8e56921c5fa8c61e3034dc8e20c.

[Mariusz Felisiak <felisiak.mariusz@…>]

In fe9c7ded:

Fixed #32200 -- Fixed grouping by ExpressionWrapper() with Q objects.

Thanks Gordon Wrigley for the report.

Regression in df32fd42b84cc6dbba173201f244491b0d154a63.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 166c0d24:

[3.1.x] Fixed #32200 -- Fixed grouping by ExpressionWrapper() with Q objects.

Thanks Gordon Wrigley for the report.

Regression in df32fd42b84cc6dbba173201f244491b0d154a63.

Backport of fe9c7ded2996364f853c524b4421274717d89d5f from master