Queryset evaluation fails with mix of nested and flattened prefetches (AttributeError on RelatedManager)

[Anthony Leontiev]

Under certain circumstances, querysets with nested prefetches fail to evaluate in Django 1.10.

With these models (populated with data):

#  A <-> B <-- C --> D
#        \__________/


class A(models.Model):
    pass


class B(models.Model):
    a = models.OneToOneField('A', null=True, related_name='b')
    d = models.ManyToManyField('D', through='C', related_name='b')


class C(models.Model):
    b = models.ForeignKey('B', related_name='c')
    d = models.ForeignKey('D', related_name='c')


class D(models.Model):
    pass

... and this query/evaluation:

    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch('c__d')
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk

Django throws this exception:

  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._prefetch_related_objects()
  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 663, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, *self._prefetch_related_lookups)
  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 1460, in prefetch_related_objects
    (through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'd' on RelatedManager object, 'b__c__d' is an invalid parameter to prefetch_related()

Interestingly enough, this practically identical variant of the above query with a fully "nested" prefetch structure succeeds:

    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch(
                    'c',
                    queryset=C.objects.all().prefetch_related(
                        'd'
                    )
                )
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk

To reproduce, download the attachment and run:

pip install -r requirements.txt
python manage.py migrate
python manage.py runserver

... and navigate to {localhost:8000/flat} and {localhost:8000/nested}.

This only seems to impact Django 1.10; the bug does not reproduce if the Django version in the {requirements.txt} file is changed to 1.9 or 1.8.

Discovered in ​https://github.com/AltSchool/dynamic-rest/pull/147
Similar to https://code.djangoproject.com/ticket/26801

[Anthony Leontiev]

test project to reproduce the bug

[Anthony Leontiev]

Discovered in an API integration test that relies on nested prefetching during an upgrade of Django 1.7 to Django 1.10.

[Simon Charette]

I haven't tried reproducing myself but the report is complete enough that I trust the developer's judgment.

Anthony, can you confirm this is also an issue when running A.objects.prefetch_related('b__c__d'')? What about B.objects.prefetch_related('c__d')?

​Bisecting the exact commit that introduced the regression would also be useful.

[vimarshc]

Hey guys,

My first time posting here.

I tried reproducing the bug, and was able to.
I added the following cases as per Simon Charette's suggestion and Django didn't throw any exception.

views.py

#Code in orinigal test project by Anthony Leontiev
def setup():
    a = A.objects.create()
    b = B.objects.create()

    b.a = a
    b.save()
    
    d = D.objects.create()
    C.objects.create(b=b, d=d)


#Succeeded: As reported above. 
def nested_prefetch(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch(
                    'c',
                    queryset=C.objects.all().prefetch_related(
                        'd'
                    )
                )
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#Failed: As reported above. 
def flat_prefetch(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch('c__d')
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#The following three cases were added by me
#Failed: As expected. 
def flat_prefetch_one(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related('c__d')
        )
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#Succeeded: I  was expecting this and the following case to fail too, but they didn't. 
def flat_prefetch_two(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch('b__c__d')
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#succeeded
def flat_prefetch_three(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related('b__c__d')
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

[vimarshc]

test cases for bug

[vimarshc]

Hey,

Just finished "bisecting the exact commit which introduced the regression".
Posting my results here.

Git output:

bdbe50a491ca41e7d4ebace47bfe8abe50a58211 is the first bad commit
commit bdbe50a491ca41e7d4ebace47bfe8abe50a58211
Author: François Freitag <francois.freitag@polyconseil.fr>
Date:   Sun Jan 10 17:54:57 2016 +0100

    Fixed #25546 -- Prevented duplicate queries with nested prefetch_related().

:040000 040000 5ad2af54d2f98566bbfdc23e04493e4c98262568 f3488a5f680bc5d4b18249aed299b6abd5f86de9 M	django
:040000 040000 59faff56062b7a347956e24315e12beeb5c7d704 c4a686067f06b7056dafffa1013bfe3c018c5e52 M	tests
bisect run success

I'll start trying to locate and fix the bug. I'll request the bug be assigned to me once I'm confident I can resolve the issue.
I'm attaching a directory with the models and tests inside it.
This directory was inside the tests directory.
I've written three test cases out of which 2 are passing and one is failing. (the one that is failing is "test_case_for_nested_and_flat_prefetches")

​Here is a link to the django repo I forked and I've pushed the aforementioned tests here too if someone wants to get them from here.
They're on the nested-bug-1.10 branch.

[Simon Charette]

#27744 was a duplicate.

[François Freitag]

​PR

[Pascal Briet]

Any update on this issue ?
It prevents us from migrating to 1.10

Thanks !

[Tim Graham]

The ticket wasn't tagged as a regression (release blocker) so unfortunately we missed the time to incorporate the fix in 1.10 which is now receiving only security updates. I'll backport the fix to 1.11.

[Tim Graham <timograham@…>]

In c0a2b950:

Fixed #27554 -- Fixed prefetch_related() crash when fetching relations in nested Prefetches.

[Tim Graham <timograham@…>]

In c679ac04:

[1.11.x] Fixed #27554 -- Fixed prefetch_related() crash when fetching relations in nested Prefetches.

Backport of c0a2b9508aa2c6eaa22e9787010276edca887f0b from master