QuerySet.values_list() combined with .extra() may produce wrong .union()

[master]

This is a revival of #29229, as a regression introduced in 2.0.5 and 1.11.13 by #29286.

Please refer to the demo code in the description of #29229.

[Carlton Gibson]

Please don't deliberately open duplicate tickets.

If there is still an issue on #29229, please comment there.

See the test case introduced in a0c03c62a8ac586e5be5b21393c925afa581efaf. Can you adjust this to demonstrate the issue you are still seeing? If so, we can re-open #29229 and address this.

(Either way we need some information to go on as to how to replicate the issue you are seeing. As far as we were aware the issue in the original description was already resolved...)

[Tim Graham]

Since a fix for that ticket has been released, it's appropriate to open a new ticket. It seems that the test that was added in the patch doesn't precisely reflect the steps to reproduce.

[master]

(I wasn't sure and I don't mind to reopen a previous ticket, but my reasoning to open a new ticket was exactly the one of Tim).

So why didn't the test case (test_union_with_two_annotated_values_list) fail after the changes of #29286? Because it is written with annotate() instead of extra(). To reproduce the steps of the demo code of #29229, replace:

qs1 = Number.objects.filter(num=1).annotate(count=Value(0, IntegerField()),).values_list('num', 'count')

by:

qs1 = Number.objects.filter(num=1).extra(select={'count': 0}).values_list('num', 'count')

Note: I just discovered that the usage of extra() is discouraged, and I will consider a replacement, but not as a priority.

My suggestion for a fix in django\db\models\sql\compiler.py\get_combinator_sql(), to work both with extra and annotate, is something of this kind:

if (not compiler.query.values_select and not compiler.query.annotations
and (self.query.values_select or self.query.annotation_select or self.query.extra_select)):
    compiler.query.set_values(tuple(self.query.values_select) +
            tuple(self.query.annotation_select) + tuple(self.query.extra_select)
    )

I'm not sure if we need the additional condition "and not compiler.query.<extra>", as I don't know what <extra> would be amongst: "extra", "extra_select" and "_extra".

[Tim Graham]

Since this was a regression, I'm open to fixing it on master and perhaps stable/2.1.x, however, be aware that we're no longer fixing bugs with QuerySet.extra() per ​discussion on django-developers.

[Mariusz Felisiak]

​PR

[GitHub <noreply@…>]

In 39461a8:

Fixed #29694 -- Fixed column mismatch crash with QuerySet.values() or values_list() after combining querysets with extra() with union(), difference(), or intersection().

Regression in 0b66c3b442875627fa6daef4ac1e90900d74290b.

[Mariusz Felisiak <felisiak.mariusz@…>]

In e7acd99:

[2.1.x] Fixed #29694 -- Fixed column mismatch crash with QuerySet.values() or values_list() after combining querysets with extra() with union(), difference(), or intersection().

Regression in 0b66c3b442875627fa6daef4ac1e90900d74290b.
Backport of 39461a83c33f0cfe719d3b139413d1f5d1e75d5e from master