id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
26318	Unexpected / duplicated queries on nested Prefetch queryset with repeated model	Anthony Leontiev	nobody	"We discovered an issue in {{{django.db.models.query.Prefetch}}} logic that results in duplicate queries made when the leaves of certain prefetch trees are accessed.

With these models:

{{{
from django.db import models


class Publisher(models.Model):
    name = models.CharField(max_length=128)


class Author(models.Model):
    name = models.CharField(max_length=128)
    publishers = models.ManyToManyField('Publisher', related_name='authors')


class Book(models.Model):
    name = models.CharField(max_length=128)
    publisher = models.ForeignKey('Publisher', related_name='books')
}}}

The following test fails:

{{{
from django.db.models.query import Prefetch
from django.test import TestCase
from .models import Author, Book, Publisher


def flatten(ls):
    return list([i for s in ls for i in s])


class PrefetchTestCase(TestCase):

    def setUp(self):
        publisher = Publisher.objects.create(name='Publisher0')
        Book.objects.create(name='Book0', publisher=publisher)
        author = Author.objects.create(name='Author0')
        author.publishers.add(publisher)

    def test_prefetch_nested(self):
        publishers = Publisher.objects.prefetch_related(
            Prefetch(
                'books',
                Book.objects.all().prefetch_related(
                    Prefetch(
                        'publisher',
                        Publisher.objects.all().prefetch_related('authors')
                    )
                )
            )
        )
        with self.assertNumQueries(4):
            publishers = list(publishers)

        with self.assertNumQueries(0):
            books = flatten([p.books.all() for p in publishers])
        with self.assertNumQueries(0):
            publishers = [b.publisher for b in books]
        with self.assertNumQueries(0):
            authors = flatten([p.authors.all() for p in publishers])
}}}

For more details (comments, queries executed) and an analogous green test-case that uses the flat prefetch form to prefetch the same tree, see the attached test package.

To run the tests:

{{{
	tar -zxvf prefetch-bug-test.tar.gz
	cd prefetch-bug-test
	make test
}}}

This issue seemed very similar to https://code.djangoproject.com/ticket/25546, but unfortunately the patch for that ticket (https://code.djangoproject.com/changeset/bdbe50a491ca41e7d4ebace47bfe8abe50a58211) did not fix this problem.

Tested in Django 1.8, 1.9, and master."	Bug	closed	Database layer (models, ORM)	dev	Normal	duplicate	queryset prefetch_related duplicate		Accepted	0	0	0	0	0	0