QuerySet.values() and values_list() for union(), difference(), and intersection() queries fails with annotated subquery

[elliott-omosheye]

Added failing test to demonstrate

    def test_union_values_subquery(self):
        items = Item.objects.filter(creator=OuterRef("pk"))
        item_authors = Author.objects.annotate(is_creator=Exists(items)).order_by()
        reports = Report.objects.filter(creator=OuterRef("pk"))
        report_authors = Author.objects.annotate(is_creator=Exists(reports)).order_by()
        all_authors = item_authors.union(report_authors).order_by()
        self.assertEqual(list(all_authors.values_list("is_creator", flat=True)), [True, True, True, True])

silently messes up the data on values/values_list() with an field

FAIL: test_union_values_subquery (queries.tests.Queries1Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/usr/src/widget_app/tests/queries/tests.py", line 95, in test_union_values_subquery
    self.assertEqual(list(all_authors.values_list("is_creator", flat=True)), [True, True, True, True])
  File "/usr/local/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/lib/python2.7/unittest/case.py", line 742, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/usr/local/lib/python2.7/unittest/case.py", line 724, in assertSequenceEqual
    self.fail(msg)
  File "/usr/local/lib/python2.7/unittest/case.py", line 410, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: [True, True, 2, 2, 3, 4, 4] != [True, True, True, True]

First differing element 2:
2
True

First list contains 3 additional elements.
First extra element 4:
3

- [True, True, 2, 2, 3, 4, 4]
+ [True, True, True, True]

throws error on empty values()/values_list()

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/usr/src/widget_app/tests/queries/tests.py", line 95, in test_union_values_subquery
    self.assertEqual(list(all_authors.values()), [True, True, True, True])
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 106, in __iter__
    for row in compiler.results_iter(chunked_fetch=self.chunked_fetch):
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 847, in results_iter
    row = self.apply_converters(row, converters)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 830, in apply_converters
    value = row[pos]
IndexError: list index out of range

The funny thing is that the produced SQL query is perfectly valid, so it isn't a case of me asking the ORM to do something especially funky. In any case if this is an unsupported action then the ORM should tell you and not produce runtime errors or silently return bad data.

[Simon Charette]

Would it be possible to provide your simplified model definition and data setup procedure as well as it makes it really hard to reproduce without them.

[elliott-omosheye]

Replying to Simon Charette:

Would it be possible to provide your simplified model definition and data setup procedure as well as it makes it really hard to reproduce without them.

Sorry for not being clearer, I am using the model definitions from queries.tests.Queries1Tests ​https://github.com/django/django/blob/stable/1.11.x/tests/queries/tests.py

[Sergey Fedoseev]

Probably duplicate of #28553.