Opened 9 months ago

Closed 9 months ago

Last modified 9 months ago

#34803 closed Bug (fixed)

Nested OuterRef crashes with AttributeError

Reported by: Pierre-Nicolas Rigal Owned by: willzhao
Component: Database layer (models, ORM) Version: 4.2
Severity: Release blocker Keywords:
Cc: Simon Charette Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Pierre-Nicolas Rigal)

Porting our application from Django 3 to 4, we're seeing exception raised in complex queries using nested OuterRef

File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1436, in filter
    return self._filter_or_exclude(False, args, kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1454, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1461, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1545, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1576, in _add_q
    child_clause, needed_inner = self.build_filter(
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1435, in build_filter
    value = self.resolve_lookup_value(value, can_reuse, allow_joins)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1204, in resolve_lookup_value
    value = value.resolve_expression(
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1923, in resolve_expression
    query = self.query.resolve_expression(*args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1145, in resolve_expression
    clone.where.resolve_expression(query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 278, in resolve_expression
    clone._resolve_node(clone, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 270, in _resolve_node
    cls._resolve_node(child, query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 274, in _resolve_node
    node.rhs = cls._resolve_leaf(node.rhs, query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 263, in _resolve_leaf
    expr = expr.resolve_expression(query, *args, **kwargs)
  File "/usr/local/python/lib/python3.10/site-packages/django/db/models/expressions.py", line 862, in resolve_expression
    if col.contains_over_clause:
AttributeError: 'OuterRef' object has no attribute 'contains_over_clause'

Looking at code, the issue seems to be the following:

class ResolvedOuterRef(F):
    """
    An object that contains a reference to an outer query.
 
    In this case, the reference to the outer query has been resolved because
    the inner query has been used as a subquery.
    """
 
    contains_aggregate = False
    contains_over_clause = False
 
    def as_sql(self, *args, **kwargs):
        raise ValueError(
            "This queryset contains a reference to an outer query and may "
            "only be used in a subquery."
        )
 
    def resolve_expression(self, *args, **kwargs):
        col = super().resolve_expression(*args, **kwargs)
        if col.contains_over_clause:
            raise NotSupportedError(
                f"Referencing outer query window expression is not supported: "
                f"{self.name}."
            )

In case of OuterRef(OuterRef( "field" )), col = super().resolve_expression(*args, **kwargs) will use:

class OuterRef(F):
    contains_aggregate = False


    def resolve_expression(self, *args, **kwargs):
        if isinstance(self.name, self.__class__):
            return self.name
        return ResolvedOuterRef(self.name)

so self.name is an OuterRef, then it returns it directly, and then if col.contains_over_clause: fails because col.contains_over_clause is not defined for OuterRef.

Looks like adding col.contains_over_clause=False to class OuterRef solves the issue - or checking if object col.contains_over_clause.

Attachments (1)

bug34803.zip (12.7 KB ) - added by Pierre-Nicolas Rigal 9 months ago.
Test case

Download all attachments as: .zip

Change History (14)

comment:1 by Pierre-Nicolas Rigal, 9 months ago

Description: modified (diff)

comment:2 by David Sanders, 9 months ago

Hi pierrenicolasr, thanks for the report!

You wouldn't happen to have a minimal example to help clarify for folks by any chance?

comment:3 by Natalia Bidart, 9 months ago

Resolution: worksforme
Status: newclosed

Hello pierrenicolasr! As mentioned by David in the previous comment, we would need a minimal Django project or a test case to reproduce this issue. I see many tests in the Django test suite that are exercising nested OuterRef calls and they are all passing:

$ grep -nr "OuterRef(OuterRef" tests/
tests/lookup/tests.py:1303:                        Author.objects.filter(alias=OuterRef(OuterRef("name")))
tests/expressions/tests.py:675:            time=OuterRef(OuterRef("time")), pk=OuterRef("start")
tests/expressions/tests.py:688:        inner = SimulationRun.objects.filter(start=OuterRef(OuterRef("pk"))).values(
tests/expressions/tests.py:832:            lastname__startswith=Left(OuterRef(OuterRef("lastname")), 1),
tests/expressions/tests.py:861:            outer_lastname=OuterRef(OuterRef("lastname")),
tests/aggregation/test_filter_argument.py:171:                            book_contact_set=OuterRef(OuterRef("pk")),
tests/aggregation/tests.py:1672:                    name=OuterRef(OuterRef("publisher__name")),

So this error seems specific of the complex queries you mention. I'll mark as worksforme for now but happy to reopen if you can provide a reproducer. Thanks!

comment:4 by Pierre-Nicolas Rigal, 9 months ago

Yes, it does not always fail - it's mainly in one complex query.
I'll see if I can create a small example reproducing the issue - thanks for the feedback !

comment:5 by Pierre-Nicolas Rigal, 9 months ago

Resolution: worksforme
Status: closednew

Obviously not real life example, but reproduces the issue:

from django.db import models


class A(models.Model):
    key = models.IntegerField()


class B(models.Model):
    a = models.ForeignKey(A, on_delete=models.CASCADE)


class C(models.Model):
    b = models.ForeignKey(B, on_delete=models.CASCADE)


class D(models.Model):
    c = models.ForeignKey(C, on_delete=models.CASCADE)
    key = models.IntegerField()

and test is:

from django.db.models import Exists, OuterRef
from django.test import TestCase

from bg34803.models import B, A, C, D


class TestOuterRef(TestCase):
    def test_nested_annotated_outerref(self):
        query = (
            A.objects
            .filter(Exists(
                B.objects
                .filter(Exists(
                    C.objects
                    .annotate(a_key=OuterRef(OuterRef('key')))
                    .filter(Exists(
                        D.objects
                        .filter(key=OuterRef('a_key'))
                    ))
                ))
            ))
        )

result:

$ python manage.py test
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_nested_annotated_outerref (bg34803.tests.TestOuterRef)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pnr/Desktop/django_issue/bug34803/bg34803/tests.py", line 14, in test_nested_annotated_outerref
    C.objects
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1436, in filter
    return self._filter_or_exclude(False, args, kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1454, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1461, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1545, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1576, in _add_q
    child_clause, needed_inner = self.build_filter(
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1417, in build_filter
    condition = filter_expr.resolve_expression(
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 278, in resolve_expression
    [
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 279, in <listcomp>
    expr.resolve_expression(query, allow_joins, reuse, summarize)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1145, in resolve_expression
    clone.where.resolve_expression(query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 278, in resolve_expression
    clone._resolve_node(clone, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 270, in _resolve_node
    cls._resolve_node(child, query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 274, in _resolve_node
    node.rhs = cls._resolve_leaf(node.rhs, query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 263, in _resolve_leaf
    expr = expr.resolve_expression(query, *args, **kwargs)
  File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 862, in resolve_expression
    if col.contains_over_clause:
AttributeError: 'OuterRef' object has no attribute 'contains_over_clause'

----------------------------------------------------------------------

We seem to have a fix by simplifying the query we use, but still wanted to report as this is a behavior change with Django 3.
Thanks.

by Pierre-Nicolas Rigal, 9 months ago

Attachment: bug34803.zip added

Test case

comment:6 by Mariusz Felisiak, 9 months ago

Cc: Simon Charette added
Severity: NormalRelease blocker
Summary: Nested OuterRef can raise AttributeError: 'OuterRef' object has no attribute 'contains_over_clause'Nested OuterRef crashes with AttributeError
Triage Stage: UnreviewedAccepted

Thanks for the report!

Regression in c67ea79aa981ae82595d89f8018a41fcd842e7c9 (backported in fc15d11f2eb26fe3d5c946e69223880bfe53e92b).

comment:7 by David Sanders, 9 months ago

@Simon I see what you're talking about now with defensive polymorphism!

comment:8 by Simon Charette, 9 months ago

If anyone wants to take a shot at this one it should be straightforward.

Defining OuterRef and ResolvedOuterRef.contains_over_clause = False like we did with .contains_aggregate in 2a431db0f5e91110b4fda05949de1f158a20ec5b and ed6b14d4591e536985222b61cb8b83908d58140d for #28621.

comment:9 by willzhao, 9 months ago

Owner: changed from nobody to willzhao
Status: newassigned

willing to fix this issue.

comment:11 by Mariusz Felisiak, 9 months ago

Triage Stage: AcceptedReady for checkin

comment:12 by Mariusz Felisiak <felisiak.mariusz@…>, 9 months ago

Resolution: fixed
Status: assignedclosed

In 9cc0d7f:

Fixed #34803 -- Fixed queryset crash when filtering againts deeply nested OuterRef annotations.

Thanks Pierre-Nicolas Rigal for the report.

Regression in c67ea79aa981ae82595d89f8018a41fcd842e7c9.

comment:13 by Mariusz Felisiak <felisiak.mariusz@…>, 9 months ago

In acfb427:

[4.2.x] Fixed #34803 -- Fixed queryset crash when filtering againts deeply nested OuterRef annotations.

Thanks Pierre-Nicolas Rigal for the report.

Regression in c67ea79aa981ae82595d89f8018a41fcd842e7c9.

Backport of 9cc0d7f7f85cecc3ad15bbc471fe6a08e4f515b6 from main

Note: See TracTickets for help on using tickets.
Back to Top