#34803 closed Bug (fixed)
Nested OuterRef crashes with AttributeError
Reported by: | Pierre-Nicolas Rigal | Owned by: | willzhao |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | 4.2 |
Severity: | Release blocker | Keywords: | |
Cc: | Simon Charette | Triage Stage: | Ready for checkin |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
Porting our application from Django 3 to 4, we're seeing exception raised in complex queries using nested OuterRef
File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1436, in filter return self._filter_or_exclude(False, args, kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1454, in _filter_or_exclude clone._filter_or_exclude_inplace(negate, args, kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1461, in _filter_or_exclude_inplace self._query.add_q(Q(*args, **kwargs)) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1545, in add_q clause, _ = self._add_q(q_object, self.used_aliases) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1576, in _add_q child_clause, needed_inner = self.build_filter( File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1435, in build_filter value = self.resolve_lookup_value(value, can_reuse, allow_joins) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1204, in resolve_lookup_value value = value.resolve_expression( File "/usr/local/python/lib/python3.10/site-packages/django/db/models/query.py", line 1923, in resolve_expression query = self.query.resolve_expression(*args, **kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1145, in resolve_expression clone.where.resolve_expression(query, *args, **kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 278, in resolve_expression clone._resolve_node(clone, *args, **kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 270, in _resolve_node cls._resolve_node(child, query, *args, **kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 274, in _resolve_node node.rhs = cls._resolve_leaf(node.rhs, query, *args, **kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/sql/where.py", line 263, in _resolve_leaf expr = expr.resolve_expression(query, *args, **kwargs) File "/usr/local/python/lib/python3.10/site-packages/django/db/models/expressions.py", line 862, in resolve_expression if col.contains_over_clause: AttributeError: 'OuterRef' object has no attribute 'contains_over_clause'
Looking at code, the issue seems to be the following:
class ResolvedOuterRef(F): """ An object that contains a reference to an outer query. In this case, the reference to the outer query has been resolved because the inner query has been used as a subquery. """ contains_aggregate = False contains_over_clause = False def as_sql(self, *args, **kwargs): raise ValueError( "This queryset contains a reference to an outer query and may " "only be used in a subquery." ) def resolve_expression(self, *args, **kwargs): col = super().resolve_expression(*args, **kwargs) if col.contains_over_clause: raise NotSupportedError( f"Referencing outer query window expression is not supported: " f"{self.name}." )
In case of OuterRef(OuterRef( "field" )), col = super().resolve_expression(*args, **kwargs)
will use:
class OuterRef(F): contains_aggregate = False def resolve_expression(self, *args, **kwargs): if isinstance(self.name, self.__class__): return self.name return ResolvedOuterRef(self.name)
so self.name is an OuterRef, then it returns it directly, and then if col.contains_over_clause:
fails because col.contains_over_clause is not defined for OuterRef.
Looks like adding col.contains_over_clause=False to class OuterRef solves the issue - or checking if object col.contains_over_clause.
Attachments (1)
Change History (14)
comment:1 by , 13 months ago
Description: | modified (diff) |
---|
comment:2 by , 13 months ago
comment:3 by , 13 months ago
Resolution: | → worksforme |
---|---|
Status: | new → closed |
Hello pierrenicolasr! As mentioned by David in the previous comment, we would need a minimal Django project or a test case to reproduce this issue. I see many tests in the Django test suite that are exercising nested OuterRef
calls and they are all passing:
$ grep -nr "OuterRef(OuterRef" tests/ tests/lookup/tests.py:1303: Author.objects.filter(alias=OuterRef(OuterRef("name"))) tests/expressions/tests.py:675: time=OuterRef(OuterRef("time")), pk=OuterRef("start") tests/expressions/tests.py:688: inner = SimulationRun.objects.filter(start=OuterRef(OuterRef("pk"))).values( tests/expressions/tests.py:832: lastname__startswith=Left(OuterRef(OuterRef("lastname")), 1), tests/expressions/tests.py:861: outer_lastname=OuterRef(OuterRef("lastname")), tests/aggregation/test_filter_argument.py:171: book_contact_set=OuterRef(OuterRef("pk")), tests/aggregation/tests.py:1672: name=OuterRef(OuterRef("publisher__name")),
So this error seems specific of the complex queries you mention. I'll mark as worksforme
for now but happy to reopen if you can provide a reproducer. Thanks!
comment:4 by , 13 months ago
Yes, it does not always fail - it's mainly in one complex query.
I'll see if I can create a small example reproducing the issue - thanks for the feedback !
comment:5 by , 13 months ago
Resolution: | worksforme |
---|---|
Status: | closed → new |
Obviously not real life example, but reproduces the issue:
from django.db import models class A(models.Model): key = models.IntegerField() class B(models.Model): a = models.ForeignKey(A, on_delete=models.CASCADE) class C(models.Model): b = models.ForeignKey(B, on_delete=models.CASCADE) class D(models.Model): c = models.ForeignKey(C, on_delete=models.CASCADE) key = models.IntegerField()
and test is:
from django.db.models import Exists, OuterRef from django.test import TestCase from bg34803.models import B, A, C, D class TestOuterRef(TestCase): def test_nested_annotated_outerref(self): query = ( A.objects .filter(Exists( B.objects .filter(Exists( C.objects .annotate(a_key=OuterRef(OuterRef('key'))) .filter(Exists( D.objects .filter(key=OuterRef('a_key')) )) )) )) )
result:
$ python manage.py test Found 1 test(s). Creating test database for alias 'default'... System check identified no issues (0 silenced). E ====================================================================== ERROR: test_nested_annotated_outerref (bg34803.tests.TestOuterRef) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/pnr/Desktop/django_issue/bug34803/bg34803/tests.py", line 14, in test_nested_annotated_outerref C.objects File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1436, in filter return self._filter_or_exclude(False, args, kwargs) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1454, in _filter_or_exclude clone._filter_or_exclude_inplace(negate, args, kwargs) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/query.py", line 1461, in _filter_or_exclude_inplace self._query.add_q(Q(*args, **kwargs)) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1545, in add_q clause, _ = self._add_q(q_object, self.used_aliases) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1576, in _add_q child_clause, needed_inner = self.build_filter( File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1417, in build_filter condition = filter_expr.resolve_expression( File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 278, in resolve_expression [ File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 279, in <listcomp> expr.resolve_expression(query, allow_joins, reuse, summarize) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 1145, in resolve_expression clone.where.resolve_expression(query, *args, **kwargs) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 278, in resolve_expression clone._resolve_node(clone, *args, **kwargs) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 270, in _resolve_node cls._resolve_node(child, query, *args, **kwargs) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 274, in _resolve_node node.rhs = cls._resolve_leaf(node.rhs, query, *args, **kwargs) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/sql/where.py", line 263, in _resolve_leaf expr = expr.resolve_expression(query, *args, **kwargs) File "/Users/pnr/Desktop/django_issue/.env/lib/python3.9/site-packages/django/db/models/expressions.py", line 862, in resolve_expression if col.contains_over_clause: AttributeError: 'OuterRef' object has no attribute 'contains_over_clause' ----------------------------------------------------------------------
We seem to have a fix by simplifying the query we use, but still wanted to report as this is a behavior change with Django 3.
Thanks.
comment:6 by , 13 months ago
Cc: | added |
---|---|
Severity: | Normal → Release blocker |
Summary: | Nested OuterRef can raise AttributeError: 'OuterRef' object has no attribute 'contains_over_clause' → Nested OuterRef crashes with AttributeError |
Triage Stage: | Unreviewed → Accepted |
Thanks for the report!
Regression in c67ea79aa981ae82595d89f8018a41fcd842e7c9 (backported in fc15d11f2eb26fe3d5c946e69223880bfe53e92b).
comment:7 by , 13 months ago
@Simon I see what you're talking about now with defensive polymorphism!
comment:8 by , 13 months ago
If anyone wants to take a shot at this one it should be straightforward.
Defining OuterRef
and ResolvedOuterRef.contains_over_clause = False
like we did with .contains_aggregate
in 2a431db0f5e91110b4fda05949de1f158a20ec5b and ed6b14d4591e536985222b61cb8b83908d58140d for #28621.
comment:9 by , 13 months ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
willing to fix this issue.
comment:11 by , 13 months ago
Triage Stage: | Accepted → Ready for checkin |
---|
Hi pierrenicolasr, thanks for the report!
You wouldn't happen to have a minimal example to help clarify for folks by any chance?