id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
16409	`defer()` and `only()` don't play nice with `annotate()`	Tai Lee	nobody	"When adding `defer()` or `only()` to an existing query that used `annotate()` to cut out some fields, I started getting IndexError exceptions.

At first, I thought it was due to `aggregate_start` not being reduced by the number of deferred fields in `QuerySet.iterator()`, but when I looked at the SQL being generated I saw that ALL fields were being dropped from the query, except for the annotation. I suspect that even if this is fixed, `aggregate_start` will also need to be reduced by the number of deferred fields.

Here's the traceback:

{{{
>>> from django.db.models import Count
>>> from django.contrib.auth.models import *
>>> from django.contrib.admin.models import *
>>> User.objects.annotate(Count('logentry'))
[<User: manager>, <User: trak:1:10>, <User: admin>, <User: trak:3:5>, <User: trak:1:24>, <User: trak:3:9>, <User: trak:1:3>, <User: trak:4:11>, <User: trak:5:21>, <User: trak:1:14>, <User: trak:1:15>, <User: trak:1:2>, <User: trak:4:18>, <User: trak:4:19>, <User: trak:5:17>, <User: trak:3:6>, <User: trak:1:23>, <User: trak:3:16>, <User: trak:5:22>, <User: trak:4:12>, '...(remaining elements truncated)...']
>>> User.objects.annotate(Count('logentry')).defer('first_name')
Traceback (most recent call last):
  File ""<console>"", line 1, in <module>
  File ""/Users/mrmachine/myproject/django/db/models/query.py"", line 69, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File ""/Users/mrmachine/myproject/django/db/models/query.py"", line 84, in __len__
    self._result_cache.extend(self._iter)
  File ""/Users/mrmachine/myproject/django/db/models/query.py"", line 300, in iterator
    setattr(obj, aggregate, row[i+aggregate_start])
IndexError: tuple index out of range
}}}

Here's the generated SQL:

{{{
>>> str(User.objects.annotate(Count('logentry')).query)
'SELECT ""auth_user"".""id"", ""auth_user"".""username"", ""auth_user"".""first_name"", ""auth_user"".""last_name"", ""auth_user"".""email"", ""auth_user"".""password"", ""auth_user"".""is_staff"", ""auth_user"".""is_active"", ""auth_user"".""is_superuser"", ""auth_user"".""last_login"", ""auth_user"".""date_joined"", COUNT(""django_admin_log"".""id"") AS ""logentry__count"" FROM ""auth_user"" LEFT OUTER JOIN ""django_admin_log"" ON (""auth_user"".""id"" = ""django_admin_log"".""user_id"") GROUP BY ""auth_user"".""id"", ""auth_user"".""username"", ""auth_user"".""first_name"", ""auth_user"".""last_name"", ""auth_user"".""email"", ""auth_user"".""password"", ""auth_user"".""is_staff"", ""auth_user"".""is_active"", ""auth_user"".""is_superuser"", ""auth_user"".""last_login"", ""auth_user"".""date_joined""'
>>> str(User.objects.annotate(Count('logentry')).defer('first_name').query)
'SELECT COUNT(""django_admin_log"".""id"") AS ""logentry__count"" FROM ""auth_user"" LEFT OUTER JOIN ""django_admin_log"" ON (""auth_user"".""id"" = ""django_admin_log"".""user_id"") GROUP BY ""auth_user"".""id"", ""auth_user"".""username"", ""auth_user"".""first_name"", ""auth_user"".""last_name"", ""auth_user"".""email"", ""auth_user"".""password"", ""auth_user"".""is_staff"", ""auth_user"".""is_active"", ""auth_user"".""is_superuser"", ""auth_user"".""last_login"", ""auth_user"".""date_joined""'
}}}
"	Bug	closed	GIS	dev	Normal	fixed	annotate defer only count	petr.gorodechnyj@…	Accepted	1	0	0	0	0	0