Context Navigation

← Previous Ticket
Next Ticket →

#34803 closed Bug (fixed)

Nested OuterRef crashes with AttributeError

Reported by:	Pierre-Nicolas Rigal	Owned by:	willzhao
Component:	Database layer (models, ORM)	Version:	4.2
Severity:	Release blocker	Keywords:
Cc:	Simon Charette	Triage Stage:	Ready for checkin
Has patch:	yes	Needs documentation:	no
Needs tests:	no	Patch needs improvement:	no
Easy pickings:	no	UI/UX:	no

Description (last modified by Pierre-Nicolas Rigal)

Porting our application from Django 3 to 4, we're seeing exception raised in complex queries using nested OuterRef

File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1436, in filter
    return self._filter_or_exclude(False, args, kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1454, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1461, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1545, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1576, in _add_q
    child_clause, needed_inner = self.build_filter(
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1435, in build_filter
    value = self.resolve_lookup_value(value, can_reuse, allow_joins)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1204, in resolve_lookup_value
    value = value.resolve_expression(
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1923, in resolve_expression
    query = self.query.resolve_expression(*args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1145, in resolve_expression
    clone.where.resolve_expression(query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 278, in resolve_expression
    clone._resolve_node(clone, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 270, in _resolve_node
    cls._resolve_node(child, query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 274, in _resolve_node
    node.rhs = cls._resolve_leaf(node.rhs, query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 263, in _resolve_leaf
    expr = expr.resolve_expression(query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/expressions.py", line 862, in resolve_expression
    if col.contains_over_clause:
AttributeError: 'OuterRef' object has no attribute 'contains_over_clause'

Looking at code, the issue seems to be the following:

class ResolvedOuterRef(F):
    """
    An object that contains a reference to an outer query.
 
    In this case, the reference to the outer query has been resolved because
    the inner query has been used as a subquery.
    """
 
    contains_aggregate = False
    contains_over_clause = False
 
    def as_sql(self, *args, **kwargs):
        raise ValueError(
            "This queryset contains a reference to an outer query and may "
            "only be used in a subquery."
        )
 
    def resolve_expression(self, *args, **kwargs):
        col = super().resolve_expression(*args, **kwargs)
        if col.contains_over_clause:
            raise NotSupportedError(
                f"Referencing outer query window expression is not supported: "
                f"{self.name}."
            )

In case of OuterRef(OuterRef( "field" )), col = super().resolve_expression(*args, **kwargs) will use:

class OuterRef(F):
    contains_aggregate = False


    def resolve_expression(self, *args, **kwargs):
        if isinstance(self.name, self.__class__):
            return self.name
        return ResolvedOuterRef(self.name)

so self.name is an OuterRef, then it returns it directly, and then if col.contains_over_clause: fails because col.contains_over_clause is not defined for OuterRef.

Looks like adding col.contains_over_clause=False to class OuterRef solves the issue - or checking if object col.contains_over_clause.

Attachments (1)

bug34803.zip (12.7 KB ) - added by Pierre-Nicolas Rigal 2 years ago.: Test case

Download all attachments as: .zip

Change History (14)

comment:1 by Pierre-Nicolas Rigal, 2 years ago

Description:	modified (diff)

comment:2 by David Sanders, 2 years ago

Hi pierrenicolasr, thanks for the report!

You wouldn't happen to have a minimal example to help clarify for folks by any chance?

comment:3 by Natalia Bidart, 2 years ago

Resolution:	→ worksforme
Status:	new → closed

Hello pierrenicolasr! As mentioned by David in the previous comment, we would need a minimal Django project or a test case to reproduce this issue. I see many tests in the Django test suite that are exercising nested OuterRef calls and they are all passing:

$ grep -nr "OuterRef(OuterRef" tests/
tests/lookup/tests.py:1303:                        Author.objects.filter(alias=OuterRef(OuterRef("name")))
tests/expressions/tests.py:675:            time=OuterRef(OuterRef("time")), pk=OuterRef("start")
tests/expressions/tests.py:688:        inner = SimulationRun.objects.filter(start=OuterRef(OuterRef("pk"))).values(
tests/expressions/tests.py:832:            lastname__startswith=Left(OuterRef(OuterRef("lastname")), 1),
tests/expressions/tests.py:861:            outer_lastname=OuterRef(OuterRef("lastname")),
tests/aggregation/test_filter_argument.py:171:                            book_contact_set=OuterRef(OuterRef("pk")),
tests/aggregation/tests.py:1672:                    name=OuterRef(OuterRef("publisher__name")),

So this error seems specific of the complex queries you mention. I'll mark as worksforme for now but happy to reopen if you can provide a reproducer. Thanks!

comment:4 by Pierre-Nicolas Rigal, 2 years ago

Yes, it does not always fail - it's mainly in one complex query.
I'll see if I can create a small example reproducing the issue - thanks for the feedback !

comment:5 by Pierre-Nicolas Rigal, 2 years ago

Resolution:	worksforme
Status:	closed → new

Obviously not real life example, but reproduces the issue:

from django.db import models


class A(models.Model):
    key = models.IntegerField()


class B(models.Model):
    a = models.ForeignKey(A, on_delete=models.CASCADE)


class C(models.Model):
    b = models.ForeignKey(B, on_delete=models.CASCADE)


class D(models.Model):
    c = models.ForeignKey(C, on_delete=models.CASCADE)
    key = models.IntegerField()

and test is:

from django.db.models import Exists, OuterRef
from django.test import TestCase

from bg34803.models import B, A, C, D


class TestOuterRef(TestCase):
    def test_nested_annotated_outerref(self):
        query = (
            A.objects
            .filter(Exists(
                B.objects
                .filter(Exists(
                    C.objects
                    .annotate(a_key=OuterRef(OuterRef('key')))
                    .filter(Exists(
                        D.objects
                        .filter(key=OuterRef('a_key'))
                    ))
                ))
            ))
        )

result:

$ python manage.py test
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_nested_annotated_outerref (bg34803.tests.TestOuterRef)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pnr/Desktop/django_issue/bug34803/bg34803/tests.py", line 14, in test_nested_annotated_outerref
    C.objects
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1436, in filter
    return self._filter_or_exclude(False, args, kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1454, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1461, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1545, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1576, in _add_q
    child_clause, needed_inner = self.build_filter(
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1417, in build_filter
    condition = filter_expr.resolve_expression(
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 278, in resolve_expression
    [
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 279, in <listcomp>
    expr.resolve_expression(query, allow_joins, reuse, summarize)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1145, in resolve_expression
    clone.where.resolve_expression(query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 278, in resolve_expression
    clone._resolve_node(clone, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 270, in _resolve_node
    cls._resolve_node(child, query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 274, in _resolve_node
    node.rhs = cls._resolve_leaf(node.rhs, query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 263, in _resolve_leaf
    expr = expr.resolve_expression(query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 862, in resolve_expression
    if col.contains_over_clause:
AttributeError: 'OuterRef' object has no attribute 'contains_over_clause'

----------------------------------------------------------------------

We seem to have a fix by simplifying the query we use, but still wanted to report as this is a behavior change with Django 3.
Thanks.

by Pierre-Nicolas Rigal, 2 years ago

Attachment:	bug34803.zip added

Test case

comment:6 by Mariusz Felisiak, 2 years ago

Cc:	Simon Charette added
Severity:	Normal → Release blocker
Summary:	Nested OuterRef can raise AttributeError: 'OuterRef' object has no attribute 'contains_over_clause' → Nested OuterRef crashes with AttributeError
Triage Stage:	Unreviewed → Accepted