Queryset "defer" won't work properly with select_related

[ruosteinen]

Select_related and defer don't seem to work properly together. Consider following models in APP buggy.

from django.db import models

class ProblemModel(models.Model):
    name = models.CharField(max_length=100)
    target = models.ForeignKey('Target', null=True)

class Target(models.Model):
    location = models.CharField(max_length=100)

Even though Queryset API document the use of underscore in deferring, the following ends up in assertion error.

from buggy.models import Target, ProblemModel
Target.objects.all().delete(); ProblemModel.objects.all().delete()

tgt = Target.objects.create(location='Earth')
pm = ProblemModel.objects.create(name='bug', target=tgt)

pm = ProblemModel.objects.select_related('target').defer('target__location').get(name='bug')

assert pm.target.location == 'bug'

At least according to Queryset API docs, defer() is supposed to allow access to deferred fields even if they're not loaded initially.

In addition pm.target is not of type Target_Deferred_location_deferred as it should be(?). There seem to be some other weirdness as well with select_related and defer but they'll probably be fixed once nesting works properly.

This in Django 1.2 beta 1 SVN-12416

I'll start working on a patch right away.

-Tommi Penttinen-

[ruosteinen]

Seems to be related to #10733.

[Carl Meyer]

I haven't played with this yet, but your repro code doesn't look right. I would expect an assertion error with that code: pm.target.location should be 'Earth' not 'bug'.

[Russell Keith-Magee]

@carljm - Agreed: "Earth" should be the expected result. Either way, it's not working - pm.target.location is returning an empty string.

@ruosteinen - Although #10733 also deals with select_related() + defer()/only() problems, I'm not immediately convinced that it is the same problem (at least, not enough to close this as a duplicate).#10733 deals with having multiple joins to the same table; this ticket is about traversing a join correctly. I suspect the problem here is something to do with populating the related object cache correctly - the query is doing the join as it should, but because the query defers the only data field on the related object, and it is being deferred, the deferred class isn't being instantiated correctly.

[ruosteinen]

@carljm, sorry. Changed the code a bit on the fly. But as russellm said, it still doesn't work.

@russellm True. Although in the proposed patch to #10733 only_load=only_load get passed properly to django.db.models.query.get_cached_row while recursing and after that relationships with deferred fields work properly i.e. type(pm.target) == <class 'buggy.models.Target_Deferred_location'>.

[ruosteinen]

I'll mark this as "has patch" since one is available at #10733.

[Russell Keith-Magee]

(In [12817]) Fixed #12851 -- Corrected the interaction of defer() with select_related(). Thanks to ruosteinen for the report.

[Russell Keith-Magee]

(In [12818]) [1.1.X] Fixed #12851 -- Corrected the interaction of defer() with select_related(). Thanks to ruosteinen for the report.

Backport of r12817 from trunk.

[Gregor Müllegger]

The current fix for this bug does not work if you use model inheritance. Here is a failing testcase.

[Gregor Müllegger]

Unfortunatelly this bug is still valid in trunk if you use model inheritance.

My history with this bug was the following:

• Discovered it with 1.2-beta-1
• Saw that it was fixed in trunk and updated (and worked)
• Changed my models to use inheritance which broke things again.

I added a testcase which fails for me with the following backtrace:

Failed example:
    markus.value
Exception raised:
    Traceback (most recent call last):
      File "C:\Python26\lib\site-packages\django\test\_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest issue12851.models.__test__.API_TESTS[45]>", line 1, in <module>
        markus.value
      File "C:\Python26\lib\site-packages\django\db\models\query_utils.py", line 208, in __get__
        cls._base_manager.filter(pk=instance.pk).only(name).using(
      File "C:\Python26\lib\site-packages\django\db\models\manager.py", line 141, in filter
        return self.get_query_set().filter(*args, **kwargs)
      File "C:\Python26\lib\site-packages\django\db\models\query.py", line 550, in filter
        return self._filter_or_exclude(False, *args, **kwargs)
      File "C:\Python26\lib\site-packages\django\db\models\query.py", line 568, in _filter_or_exclude
        clone.query.add_q(Q(*args, **kwargs))
      File "C:\Python26\lib\site-packages\django\db\models\sql\query.py", line 1131, in add_q
        can_reuse=used_aliases)
      File "C:\Python26\lib\site-packages\django\db\models\sql\query.py", line 1071, in add_filter
        connector)
      File "C:\Python26\lib\site-packages\django\db\models\sql\where.py", line 66, in add
        value = obj.prepare(lookup_type, value)
      File "C:\Python26\lib\site-packages\django\db\models\sql\where.py", line 299, in prepare
        return self.field.get_prep_lookup(lookup_type, value)
      File "C:\Python26\lib\site-packages\django\db\models\fields\related.py", line 134, in get_prep_lookup
        return self._pk_trace(value, 'get_prep_lookup', lookup_type)
      File "C:\Python26\lib\site-packages\django\db\models\fields\related.py", line 196, in _pk_trace
        v = getattr(field, prep_func)(lookup_type, v, **kwargs)
      File "C:\Python26\lib\site-packages\django\db\models\fields\__init__.py", line 292, in get_prep_lookup
        return self.get_prep_value(value)
      File "C:\Python26\lib\site-packages\django\db\models\fields\__init__.py", line 476, in get_prep_value
        return int(value)
    ValueError: invalid literal for int() with base 10: 'Markus McDonalds'

IMO the SQL query is correct and retrieves the correct fields from the database, but the fields are populated in the wrong way.

[Russell Keith-Magee]

(In [13055]) [1.1.X] Fixed #12851 -- Corrected the loading of values when select_related() is used on inherited models. Thanks to phxx for the report and test case.

Backport of r13054 from trunk.

[Gregor Müllegger]

Here is another testcase using "only" instead of "defer" which fails with the current trunk.

[Gregor Müllegger]

I'm very sorry for reopening the ticket *again*.

But the issue is still there if you use the "only" method instead of "defer". I've attached a new testcase which fails with the current trunk.

Here is the traceback:

Failed example:
    troy.state.name
Exception raised:
    Traceback (most recent call last):
      File "C:\Python26\lib\site-packages\django\test\_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest issue12851.models.__test__.API_TESTS[56]>", line 1, in <module>
        troy.state.name
      File "C:\Python26\lib\site-packages\django\db\models\query_utils.py", line 208, in __get__
        cls._base_manager.filter(pk=instance.pk).only(name).using(
      File "C:\Python26\lib\site-packages\django\db\models\manager.py", line 141, in filter
        return self.get_query_set().filter(*args, **kwargs)
      File "C:\Python26\lib\site-packages\django\db\models\query.py", line 550, in filter
        return self._filter_or_exclude(False, *args, **kwargs)
      File "C:\Python26\lib\site-packages\django\db\models\query.py", line 568, in _filter_or_exclude
        clone.query.add_q(Q(*args, **kwargs))
      File "C:\Python26\lib\site-packages\django\db\models\sql\query.py", line 1131, in add_q
        can_reuse=used_aliases)
      File "C:\Python26\lib\site-packages\django\db\models\sql\query.py", line 1071, in add_filter
        connector)
      File "C:\Python26\lib\site-packages\django\db\models\sql\where.py", line 66, in add
        value = obj.prepare(lookup_type, value)
      File "C:\Python26\lib\site-packages\django\db\models\sql\where.py", line 299, in prepare
        return self.field.get_prep_lookup(lookup_type, value)
      File "C:\Python26\lib\site-packages\django\db\models\fields\related.py", line 134, in get_prep_lookup
        return self._pk_trace(value, 'get_prep_lookup', lookup_type)
      File "C:\Python26\lib\site-packages\django\db\models\fields\related.py", line 196, in _pk_trace
        v = getattr(field, prep_func)(lookup_type, v, **kwargs)
      File "C:\Python26\lib\site-packages\django\db\models\fields\__init__.py", line 292, in get_prep_lookup
        return self.get_prep_value(value)
      File "C:\Python26\lib\site-packages\django\db\models\fields\__init__.py", line 476, in get_prep_value
        return int(value)
    ValueError: invalid literal for int() with base 10: 'Troy Buswell'

[Russell Keith-Magee]

(In [13059]) Fixed #12851 -- Another attempt at fixing select_related() with inherited models, this time with only(). Thanks to phxx for the test case.

[Russell Keith-Magee]

(In [13060]) [1.1.X] Fixed #12851 -- Another attempt at fixing select_related() with inherited models, this time with only(). Thanks to phxx for the test case.

Backport of r13059 from trunk.

[Jacob]

Milestone 1.2 deleted