Can't defer() fields from super- and sub-class at the same time

[Jeremy Kerr]

Using the models:

from django.db import models

class Base(models.Model):
    f1 = models.CharField(max_length=10)

class Sub(Base):
    f2 = models.CharField(max_length=10)

it seems that I can't defer() both f1 and f2 in a single query:

>>> Sub.objects.defer('f1', 'f2').query.sql_with_params()[0]
u'SELECT "defer_base"."id", "defer_base"."f1", "defer_sub"."base_ptr_id" FROM "defer_sub" INNER JOIN "defer_base" ON ("defer_sub"."base_ptr_id" = "defer_base"."id")'

(we're seeing f1 in the SELECT value list).

I seem to be able to defer f1 or f2 separately though.

I'm no django hacker, but: it seems as though django.db.models.sql.query.Query.deferred_to_data() is iterating both models in the loop:

#640:
            for model, values in six.iteritems(seen):
                for field in model._meta.fields:
                    if field in values:
                        continue

- and so is discovering f1 twice: once as Base.f1 and again as Sub.f1. Since the field in values test only skips Base.f1, we're still left with Sub.f1 in the workset dict.

[Simon Charette]

I think your intuition is right and that the bug lies in deferred_to_data(). Could you try replacing for field in model._meta.fields by for field in model._meta.local_fields and see if it helps?

[Jeremy Kerr]

Replying to Simon Charette:

I think your intuition is right and that the bug lies in deferred_to_data(). Could you try replacing for field in model._meta.fields by for field in model._meta.local_fields and see if it helps?

Yep, that seems to fix the issue:

>>> Sub.objects.defer('f1', 'f2').query.sql_with_params()[0]
u'SELECT "defer_base"."id", "defer_sub"."base_ptr_id" FROM "defer_sub" INNER JOIN "defer_base" ON ("defer_sub"."base_ptr_id" = "defer_base"."id")'

I was concerned that using local_fields would omit inherited fields from the query, but looks like that's not the case; inherited (but not deferred) fields work fine with that change:

class Base(models.Model):
    f1 = models.CharField(max_length=10)
    f3 = models.CharField(max_length=10)

class Sub(Base):
    f2 = models.CharField(max_length=10)

>>> Sub.objects.defer('f1', 'f2').query.sql_with_params()[0]
u'SELECT "defer_base"."id", "defer_base"."f3", "defer_sub"."base_ptr_id" FROM "defer_sub" INNER JOIN "defer_base" ON ("defer_sub"."base_ptr_id" = "defer_base"."id")'

- correctly includes f3 in the query. Behaviour also look to remain correct when only deferring fields from one class too.

[Jeremy Kerr]

Patch coming.

[Jeremy Kerr]

Pull request sent: ​PR

Since this is a trivial patch (and my first), I've not added to AUTHORS or submitted a CLA. Let me know if I need to do either of those.

[Simon Charette]

Awesome, thank you for submitting a PR and adding a test!

[Tim Graham <timograham@…>]

In 84b7cb7d:

Fixed #28549 -- Fixed QuerySet.defer() with super and subclass fields.

Previously, deferring fields in different classes didn't omit the
superclass' deferred field.

Thanks Simon Charette for the suggested fix.

[Chris Lamb]

Curious why this didn't reach into the 1.11.6 bugfix release.? We are carrying this patch in Debian after a user request:

​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876816

(It was the cause of performance regressions in his application so seemed sufficiently important to carry)

[Simon Charette]

Chris, probably because it was never reported as a regression in the first place. Can you confirm this was working in a previous version of Django?

[Chris Lamb]

Replying to Simon Charette:

Chris, probably because it was never reported as a regression in the first place. Can you confirm this was working in a previous version of Django?

Yep. If I understand your question correctly, from: ​https://bugs.debian.org/876816#5:

This bug caused significant performance degradation when we upgraded a
Django [1.x] application to a new version that relied on model inheritance.