`defer()` and `only()` don't play nice with `annotate()`

[Tai Lee]

When adding defer() or only() to an existing query that used annotate() to cut out some fields, I started getting IndexError exceptions.

At first, I thought it was due to aggregate_start not being reduced by the number of deferred fields in QuerySet.iterator(), but when I looked at the SQL being generated I saw that ALL fields were being dropped from the query, except for the annotation. I suspect that even if this is fixed, aggregate_start will also need to be reduced by the number of deferred fields.

Here's the traceback:

>>> from django.db.models import Count
>>> from django.contrib.auth.models import *
>>> from django.contrib.admin.models import *
>>> User.objects.annotate(Count('logentry'))
[<User: manager>, <User: trak:1:10>, <User: admin>, <User: trak:3:5>, <User: trak:1:24>, <User: trak:3:9>, <User: trak:1:3>, <User: trak:4:11>, <User: trak:5:21>, <User: trak:1:14>, <User: trak:1:15>, <User: trak:1:2>, <User: trak:4:18>, <User: trak:4:19>, <User: trak:5:17>, <User: trak:3:6>, <User: trak:1:23>, <User: trak:3:16>, <User: trak:5:22>, <User: trak:4:12>, '...(remaining elements truncated)...']
>>> User.objects.annotate(Count('logentry')).defer('first_name')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/mrmachine/myproject/django/db/models/query.py", line 69, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/Users/mrmachine/myproject/django/db/models/query.py", line 84, in __len__
    self._result_cache.extend(self._iter)
  File "/Users/mrmachine/myproject/django/db/models/query.py", line 300, in iterator
    setattr(obj, aggregate, row[i+aggregate_start])
IndexError: tuple index out of range

Here's the generated SQL:

>>> str(User.objects.annotate(Count('logentry')).query)
'SELECT "auth_user"."id", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."password", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."is_superuser", "auth_user"."last_login", "auth_user"."date_joined", COUNT("django_admin_log"."id") AS "logentry__count" FROM "auth_user" LEFT OUTER JOIN "django_admin_log" ON ("auth_user"."id" = "django_admin_log"."user_id") GROUP BY "auth_user"."id", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."password", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."is_superuser", "auth_user"."last_login", "auth_user"."date_joined"'
>>> str(User.objects.annotate(Count('logentry')).defer('first_name').query)
'SELECT COUNT("django_admin_log"."id") AS "logentry__count" FROM "auth_user" LEFT OUTER JOIN "django_admin_log" ON ("auth_user"."id" = "django_admin_log"."user_id") GROUP BY "auth_user"."id", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."password", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."is_superuser", "auth_user"."last_login", "auth_user"."date_joined"'

[Tai Lee]

Added a patch with a failing test case.

Creating test database for alias 'default'...
Creating test database for alias 'other'...
E
======================================================================
ERROR: test_defer (modeltests.defer.tests.DeferTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mrmachine/Subversion/django/tests/modeltests/defer/tests.py", line 141, in test_defer
    self.assertEqual(len(Secondary.objects.annotate(Count('primary')).defer('first')), 1)
  File "/Users/mrmachine/Subversion/django/django/db/models/query.py", line 82, in __len__
    self._result_cache = list(self.iterator())
  File "/Users/mrmachine/Subversion/django/django/db/models/query.py", line 300, in iterator
    setattr(obj, aggregate, row[i+aggregate_start])
IndexError: tuple index out of range

----------------------------------------------------------------------
Ran 1 test in 0.120s

FAILED (errors=1)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

[Tai Lee]

Updated patch to include a fix. Hopefully good to go.

[Tai Lee]

Failing test case and fix.

[Tai Lee]

Updated tests to use assertIsInstance. The test checks that the queryset can be successfully iterated through without raising an exception by converting it to a list.

[Ramiro Morales]

In [16522]:

Fixed #16409 -- Fixed an error condition when using QuerySet only()/defer() on the result of an annotate() call. Thanks jaklaassen AT gmail DOT com and Tai Lee for the reports and Tai for the patch.

[Jacob]

Milestone 1.4 deleted

[Petr Gorodechnyj <petr.gorodechnyj@…>]

OK, you did that all right, but now do the same for the GeoQuerySet please. If I call GeoManager's defer(), only() or values() I get the same problem that was fixed in the patch above.

[Aymeric Augustin]

Indeed — I just attached a failing test case.

[Petr Gorodechnyj <petr.gorodechnyj@…>]

Fixes this problem for GeoQuery

[Ramiro Morales]

In [17506]:

Fixed #16409 (again, this time for GeoDjango).

Thanks Aymeric Augustin for the regression test and Petr Gorodechnyj for
the patch.

Refs r16522.