Opened 9 years ago

Closed 9 years ago

#24873 closed Bug (fixed)

Error while using Prefech on more than two levels

Reported by: Gagaro Owned by: Gagaro
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: Prefetch prefetch_related
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Gagaro)

Given the following models:

# Foreign Keys

class A(models.Model):
    pass

class B(models.Model):
    a = models.ForeignKey(A, related_name='b')

class C(models.Model):
    b = models.ForeignKey(B, related_name='c')

class D(models.Model):
    c = models.ForeignKey(C, related_name='d')


#ManyToMany Fields

class E(models.Model):
    pass

class F(models.Model):
    e = models.ManyToManyField(E, related_name='f')

class G(models.Model):
    f = models.ManyToManyField(F, related_name='g')

class H(models.Model):
    g = models.ManyToManyField(G, related_name='h')

And the following tests:

class PrefetchForeignKeyTest(TestCase):
    def setUp(self):
        a = A.objects.create()
        b = B.objects.create(a=a)
        c = C.objects.create(b=b)
        D.objects.create(c=c)

    def test_with_prefetch(self):
        c = C.objects.prefetch_related(Prefetch('d'))
        b = B.objects.prefetch_related(Prefetch('c', queryset=c))
        a = A.objects.prefetch_related(Prefetch('b', queryset=b))
        list(a)
        a.first().b.first().c.first().d.first()

    def test_without_prefetch(self):
        c = C.objects.prefetch_related('d')
        b = B.objects.prefetch_related(Prefetch('c', queryset=c))
        a = A.objects.prefetch_related(Prefetch('b', queryset=b))
        list(a)
        a.first().b.first().c.first().d.first()


class PrefetchManyTest(TestCase):
    def setUp(self):
        e = E.objects.create()
        f = F.objects.create()
        g = G.objects.create()
        h = H.objects.create()
        f.e.add(e)
        g.f.add(f)
        h.g.add(g)

    def test_with_prefetch(self):
        g = G.objects.prefetch_related(Prefetch('h'))
        f = F.objects.prefetch_related(Prefetch('g', queryset=g))
        e = E.objects.prefetch_related(Prefetch('f', queryset=f))
        list(e)
        e.first().f.first().g.first().h.first()

    def test_without_prefetch(self):
        g = G.objects.prefetch_related('h')
        f = F.objects.prefetch_related(Prefetch('g', queryset=g))
        e = E.objects.prefetch_related(Prefetch('f', queryset=f))
        list(e)
        e.first().f.first().g.first().h.first()

The tests will fail. The backtrace with master is:

======================================================================
ERROR: test_with_prefetch (models_test.tests.PrefetchManyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gagaro/django/modules/test/django_test/models_test/tests.py", line 49, in test_with_prefetch
    e.first().f.first().g.first().h.first()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 543, in first
    objects = list((self if self.ordered else self.order_by('pk'))[:1])
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in _fetch_all
    self._prefetch_related_objects()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 649, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1426, in prefetch_related_objects
    (through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'f' on F object, 'f__f__g' is an invalid parameter to prefetch_related()

======================================================================
ERROR: test_without_prefetch (models_test.tests.PrefetchManyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gagaro/django/modules/test/django_test/models_test/tests.py", line 56, in test_without_prefetch
    e.first().f.first().g.first().h.first()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 543, in first
    objects = list((self if self.ordered else self.order_by('pk'))[:1])
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in _fetch_all
    self._prefetch_related_objects()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 649, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1426, in prefetch_related_objects
    (through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'f' on F object, 'f__f__g' is an invalid parameter to prefetch_related()

======================================================================
ERROR: test_with_prefetch (models_test.tests.PrefetchForeignKeyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gagaro/django/modules/test/django_test/models_test/tests.py", line 23, in test_with_prefetch
    list(a)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in _fetch_all
    self._prefetch_related_objects()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 649, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1437, in prefetch_related_objects
    obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1536, in prefetch_one_level
    prefetcher.get_prefetch_queryset(instances, lookup.get_current_queryset(level)))
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 753, in get_prefetch_queryset
    for rel_obj in queryset:
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in _fetch_all
    self._prefetch_related_objects()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 649, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1426, in prefetch_related_objects
    (through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'c' on C object, 'c__d' is an invalid parameter to prefetch_related()

======================================================================
ERROR: test_without_prefetch (models_test.tests.PrefetchForeignKeyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gagaro/django/modules/test/django_test/models_test/tests.py", line 31, in test_without_prefetch
    a.first().b.first().c.first().d.first()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 543, in first
    objects = list((self if self.ordered else self.order_by('pk'))[:1])
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in _fetch_all
    self._prefetch_related_objects()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 649, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1437, in prefetch_related_objects
    obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1536, in prefetch_one_level
    prefetcher.get_prefetch_queryset(instances, lookup.get_current_queryset(level)))
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 753, in get_prefetch_queryset
    for rel_obj in queryset:
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in _fetch_all
    self._prefetch_related_objects()
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 649, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
  File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-packages/django/db/models/query.py", line 1426, in prefetch_related_objects
    (through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'b' on B object, 'b__c' is an invalid parameter to prefetch_related()

I tested with Django 1.7, 1.8, 1.8.2 and master (commit 24718b7dccd64dfa118fe8136e7b175babec679b).

Change History (8)

comment:1 by Tim Graham, 9 years ago

Triage Stage: Unreviewed → Accepted

I don't have much experience with prefetch_related() but as far as I understand, the two tests should work identically.

comment:2 by Gagaro, 9 years ago

Description: modified (diff)

comment:3 by Gagaro, 9 years ago

I did some more testing. All the tests actually fail when I try to access the objects. I also added tests with m2m (which only fail when the objects are accessed).

comment:4 by Gagaro, 9 years ago

I found and corrected the bug. I'm working on the pull request (it may take a little bit of time as it is my first and I still need to add regression tests).

comment:5 by Gagaro, 9 years ago

Owner: changed from nobody to Gagaro
Status: new → assigned

comment:7 by Tim Graham, 9 years ago

Has patch: set
Triage Stage: Accepted → Ready for checkin

Would be nice for someone familiar with the prefetch stuff to double check this.

comment:8 by Tim Graham <timograham@…>, 9 years ago

Resolution: → fixed
Status: assigned → closed

In 74261bc5:

Fixed #24873 -- Prevented nested Prefetch objects from being overwritten.

Note: See TracTickets for help on using tickets.
Back to Top