Defer/only + annotate is broken

[jaklaassen@…]

If you use defer or only + annotate, the ORM returns an empty list. To replicate this:

class Manufacturer(models.Model):
  name = models.CharField(max_length=64)
  country = models.CharField(max_length=32)

class Car(models.Model):
  manufacturer = models.ForeignKey(Manufacturer, related_name='models')

>>>Manufacturer.objects.only('name').annotate(Count('models'))
>>>[]

This is due to a typo in django.db.models.sql.query at line 714.

The existing code:

if table in only_load and col not in only_load[table]:

should read

if table in only_load and column not in only_load[table]:

[jaklaassen@…]

django.db.models.query.py

[jaklaassen@…]

After a little more work, I had two more problems directly related to this. The iterator method of QuerySet and the results_iter method of Query seem to be calculating the index of the start of the aggregation_fields incorrectly when also using only/defer. I am a little less certain of the solution I have working for this.

I have attached the patch for Query because it is a little more involed.

QuerySet iterator() defines aggregate_start on line 204:

aggregate_start = index_start + len(self.model._meta.fields)

I would propose to define it before the existing row 257 as:

aggregate_start = index_start + len(load_fields or self.model._meta.fields)

[Alex Gaynor]

Please upload a diff against the root of the source tree, rather than a snippet of code.

[ruosteinen]

Select fields were getting garbled for only/defer fields as already discussed and index offset didn't take into accord the deferred fields.

Added a regression test as well. (Diffed on revision 12437.)

[ruosteinen]

Noticed an extra error when using select_related as well. For models:

from django.db import models
class Person(models.Model):
    first_name = models.CharField(max_length=200)
    last_name = models.CharField(max_length=50)
    best_friend = models.ForeignKey('self', blank=True, null=True, editable=False)
        
    nick_name = models.CharField(max_length=50, blank=True, null=True)

class Candy(models.Model):
    owner = models.ForeignKey(Person, related_name="candies")
    qty = models.PositiveIntegerField(default=0)


from django.db.models import Sum;

Person.objects.all().delete()
Candy.objects.all().delete()
p = Person.objects.create(first_name='Mr', last_name='Egotistic')
p.best_friend = p
p.id = 1            # For determinism
p.save()
creations = [Candy(pk=n, owner_id=p.pk, qty=42).save() for n in xrange(10)]

assert Person.objects.annotate(Sum('candies__qty')).select_related('best_friend').defer('first_name','best_friend__nick_name').get(pk=p.pk).candies__qty__sum == 378

Indexing goes wrong in django.db.sql.compiler even when applying this patch. I'll work on it but in the meantime I'll add a diff for regression.

[ruosteinen]

Regression test diff against trunk.

[Russell Keith-Magee]

Not critical for 1.2

[Ramiro Morales]

#15523 reported the same problem with slightly different symptoms.

[Ramiro Morales]

See also #16409.

[Tai Lee]

I've tried, but can't reproduce a problem with the select/extra tests in r16510. Reading the test case, I'm not sure it was valid in the first place. It creates 10 Candy objects with qty of 42, then tests that the Sum is 378 - that never would have been correct.

The issues in the original report have been fixed in [16522]. Although the symptoms changed (raising an IndexError instead of returning an empty queryset), the fix was the same. I believe the symptoms probably changed due to underlying changes in Django since this ticket was reported.

[Tai Lee]

Bogus test.

[Tai Lee]

Removed explicit PKs from test. Still bogus.

[Ramiro Morales]

I think we can safely close this as fixed in [16522]. Thanks ruosteinen and mrmachine.

[Jacob]

Milestone 1.3 deleted