Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#27554 closed Bug (fixed)

Queryset evaluation fails with mix of nested and flattened prefetches (AttributeError on RelatedManager)

Reported by: Anthony Leontiev Owned by: François Freitag
Component: Database layer (models, ORM) Version: 1.10
Severity: Release blocker Keywords: prefetch AttributeError prefetch_related nested RelatedManager
Cc: mail@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Under certain circumstances, querysets with nested prefetches fail to evaluate in Django 1.10.

With these models (populated with data):

#  A <-> B <-- C --> D
#        \__________/


class A(models.Model):
    pass


class B(models.Model):
    a = models.OneToOneField('A', null=True, related_name='b')
    d = models.ManyToManyField('D', through='C', related_name='b')


class C(models.Model):
    b = models.ForeignKey('B', related_name='c')
    d = models.ForeignKey('D', related_name='c')


class D(models.Model):
    pass

... and this query/evaluation:

    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch('c__d')
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk

Django throws this exception:

  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._prefetch_related_objects()
  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 663, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, *self._prefetch_related_lookups)
  File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 1460, in prefetch_related_objects
    (through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'd' on RelatedManager object, 'b__c__d' is an invalid parameter to prefetch_related()

Interestingly enough, this practically identical variant of the above query with a fully "nested" prefetch structure succeeds:

    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch(
                    'c',
                    queryset=C.objects.all().prefetch_related(
                        'd'
                    )
                )
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk

To reproduce, download the attachment and run:

pip install -r requirements.txt
python manage.py migrate
python manage.py runserver

... and navigate to {localhost:8000/flat} and {localhost:8000/nested}.

This only seems to impact Django 1.10; the bug does not reproduce if the Django version in the {requirements.txt} file is changed to 1.9 or 1.8.

Discovered in https://github.com/AltSchool/dynamic-rest/pull/147
Similar to https://code.djangoproject.com/ticket/26801

Attachments (2)

prefetchbug.tar.gz (5.5 KB) - added by Anthony Leontiev 3 years ago.
test project to reproduce the bug
flattened-prefetch.zip (3.3 KB) - added by vimarshc 3 years ago.
test cases for bug

Download all attachments as: .zip

Change History (13)

Changed 3 years ago by Anthony Leontiev

Attachment: prefetchbug.tar.gz added

test project to reproduce the bug

comment:1 Changed 3 years ago by Anthony Leontiev

Discovered in an API integration test that relies on nested prefetching during an upgrade of Django 1.7 to Django 1.10.

comment:2 Changed 3 years ago by Simon Charette

Triage Stage: UnreviewedAccepted

I haven't tried reproducing myself but the report is complete enough that I trust the developer's judgment.

Anthony, can you confirm this is also an issue when running A.objects.prefetch_related('b__c__d'')? What about B.objects.prefetch_related('c__d')?

Bisecting the exact commit that introduced the regression would also be useful.

Last edited 3 years ago by Tim Graham (previous) (diff)

comment:3 Changed 3 years ago by vimarshc

Hey guys,

My first time posting here.

I tried reproducing the bug, and was able to.
I added the following cases as per Simon Charette's suggestion and Django didn't throw any exception.

views.py

#Code in orinigal test project by Anthony Leontiev
def setup():
    a = A.objects.create()
    b = B.objects.create()

    b.a = a
    b.save()
    
    d = D.objects.create()
    C.objects.create(b=b, d=d)


#Succeeded: As reported above. 
def nested_prefetch(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch(
                    'c',
                    queryset=C.objects.all().prefetch_related(
                        'd'
                    )
                )
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#Failed: As reported above. 
def flat_prefetch(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related(
                Prefetch('c__d')
            )
        )
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#The following three cases were added by me
#Failed: As expected. 
def flat_prefetch_one(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch(
            'b',
            queryset=B.objects.all().prefetch_related('c__d')
        )
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#Succeeded: I  was expecting this and the following case to fail too, but they didn't. 
def flat_prefetch_two(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related(
        Prefetch('b__c__d')
    )
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

#succeeded
def flat_prefetch_three(request, *args, **kwargs):
    setup()
    queryset = A.objects.all().prefetch_related('b__c__d')
    content = queryset[0].b.c.all()[0].d.pk
    return HttpResponse(content)

Changed 3 years ago by vimarshc

Attachment: flattened-prefetch.zip added

test cases for bug

comment:4 Changed 3 years ago by vimarshc

Hey,

Just finished "bisecting the exact commit which introduced the regression".
Posting my results here.

Git output:

bdbe50a491ca41e7d4ebace47bfe8abe50a58211 is the first bad commit
commit bdbe50a491ca41e7d4ebace47bfe8abe50a58211
Author: François Freitag <francois.freitag@polyconseil.fr>
Date:   Sun Jan 10 17:54:57 2016 +0100

    Fixed #25546 -- Prevented duplicate queries with nested prefetch_related().

:040000 040000 5ad2af54d2f98566bbfdc23e04493e4c98262568 f3488a5f680bc5d4b18249aed299b6abd5f86de9 M	django
:040000 040000 59faff56062b7a347956e24315e12beeb5c7d704 c4a686067f06b7056dafffa1013bfe3c018c5e52 M	tests
bisect run success

I'll start trying to locate and fix the bug. I'll request the bug be assigned to me once I'm confident I can resolve the issue.
I'm attaching a directory with the models and tests inside it.
This directory was inside the tests directory.
I've written three test cases out of which 2 are passing and one is failing. (the one that is failing is "test_case_for_nested_and_flat_prefetches")

Here is a link to the django repo I forked and I've pushed the aforementioned tests here too if someone wants to get them from here.
They're on the nested-bug-1.10 branch.

comment:5 Changed 3 years ago by François Freitag

Cc: mail@… added

comment:6 Changed 3 years ago by Simon Charette

#27744 was a duplicate.

comment:7 Changed 3 years ago by François Freitag

Has patch: set
Owner: changed from nobody to François Freitag
Status: newassigned

comment:8 Changed 3 years ago by Pascal Briet

Any update on this issue ?
It prevents us from migrating to 1.10

Thanks !

comment:9 Changed 3 years ago by Tim Graham

Severity: NormalRelease blocker
Triage Stage: AcceptedReady for checkin

The ticket wasn't tagged as a regression (release blocker) so unfortunately we missed the time to incorporate the fix in 1.10 which is now receiving only security updates. I'll backport the fix to 1.11.

comment:10 Changed 3 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In c0a2b950:

Fixed #27554 -- Fixed prefetch_related() crash when fetching relations in nested Prefetches.

comment:11 Changed 3 years ago by Tim Graham <timograham@…>

In c679ac04:

[1.11.x] Fixed #27554 -- Fixed prefetch_related() crash when fetching relations in nested Prefetches.

Backport of c0a2b9508aa2c6eaa22e9787010276edca887f0b from master

Note: See TracTickets for help on using tickets.
Back to Top