Foreign key isn't set on object after related set access if ForeignKey uses to_field

[Carsten Fuchs]

Using the example models Blog and Entry from ​https://docs.djangoproject.com/en/2.1/topics/db/queries/#related-objects, this code:

    b = Blog.objects.get(name="...")
    for e in b.entry_set.all():
        print(e.blog)   # e.blog == b

should not trigger an access to the database, because e.blog == b.
Instead, for each e, the access to e.blog unexpectedly causes a query to fetch the blog object.

(Originally posted at ​https://groups.google.com/d/msg/django-users/Jwk9yd8Mikc/DhaEf8I5BgAJ)

[Tim Graham]

This test doesn't fail for me. Can you provide a failing test?

blog = Blog.objects.create(name='Beatles Blog')
Entry.objects.create(blog=blog)
Entry.objects.create(blog=blog)
with self.assertNumQueries(1):
    for e in blog.entry_set.all():
        self.assertEqual(e.blog.name, 'Beatles Blog')

[Carsten Fuchs]

Hi Tim,

it seems that this is triggered when the ForeignKey's to_field parameter is used.
The test fails with the following model EntryB:

from django.db import models

class Blog(models.Model):
    key = models.CharField(max_length=10, unique=True)
    name = models.CharField(max_length=100)

    def __str__(self):
        return self.name

class EntryA(models.Model):
    blog = models.ForeignKey(Blog, on_delete=models.CASCADE)
    headline = models.CharField(max_length=255)

class EntryB(models.Model):
    blog = models.ForeignKey(Blog, on_delete=models.CASCADE, to_field='key')   # this triggers the problem
    headline = models.CharField(max_length=255)

Test cases:

from django.test import TestCase
from TestApp.models import Blog, EntryA, EntryB


class TestBlogs(TestCase):

    def test_A(self):
        blog = Blog.objects.create(name='Beatles Blog')
        EntryA.objects.create(blog=blog)
        EntryA.objects.create(blog=blog)

        with self.assertNumQueries(1):    # succeeds
            for e in blog.entrya_set.all():
                self.assertEqual(e.blog.name, 'Beatles Blog')

    def test_B(self):
        blog = Blog.objects.create(name='Beatles Blog')
        EntryB.objects.create(blog=blog)
        EntryB.objects.create(blog=blog)

        with self.assertNumQueries(1):    # This fails!
            for e in blog.entryb_set.all():
                self.assertEqual(e.blog.name, 'Beatles Blog')

Test results:

$ ./manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.F
======================================================================
FAIL: test_B (TestApp.tests.TestBlogs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/carsten/xxxTestProject/TestApp/tests.py", line 23, in test_B
    self.assertEqual(e.blog.name, 'Beatles Blog')
  File "/home/carsten/.virtualenvs/Zeiterfassung/lib/python3.6/site-packages/django/test/testcases.py", line 88, in __exit__
    query['sql'] for query in self.captured_queries
AssertionError: 3 != 1 : 3 queries executed, 1 expected
Captured queries were:
SELECT "TestApp_entryb"."id", "TestApp_entryb"."blog_id", "TestApp_entryb"."headline" FROM "TestApp_entryb" WHERE "TestApp_entryb"."blog_id" = ''
SELECT "TestApp_blog"."id", "TestApp_blog"."key", "TestApp_blog"."name" FROM "TestApp_blog" WHERE "TestApp_blog"."key" = ''
SELECT "TestApp_blog"."id", "TestApp_blog"."key", "TestApp_blog"."name" FROM "TestApp_blog" WHERE "TestApp_blog"."key" = ''

----------------------------------------------------------------------
Ran 2 tests in 0.003s

FAILED (failures=1)
Destroying test database for alias 'default'...

[Tim Graham]

Confirmed the issue on master at 3d4d0a25b299a97314582156a0d63d939662d310.

[Simon Charette]

​https://github.com/django/django/pull/10595

[Tim Graham <timograham@…>]

In 75dfa92a:

Fixed #29908 -- Fixed setting of foreign key after related set access if ForeignKey uses to_field.

Adjusted known related objects handling of target fields which relies on
from and to_fields and has the side effect of fixing a bug bug causing
N+1 queries when using reverse foreign objects.

Thanks Carsten Fuchs for the report.

[Simon Charette <charette.s@…>]

In 0cf85e6b:

Refs #29908 -- Optimized known related objects assignment.

Since CPython implements a C level attrgetter(*attrs) it even outperforms the
most common case of a single known related object since the resulting attribute
values tuple is built in C.