GIS distance lookups fail within subqueries using OuterRef

[Andrew Brown]

I discovered this when trying to make a query of this form:

from django.db import models
from django.contrib.gis.db.models.fields import PointField, RasterField

class ModelA(models.Model):
    pointA = PointField()

class ModelB(models.Model):
    pointB = PointField()


def query1():
    return ModelA.objects.annotate(
        has_value=Exists(ModelB.objects.filter(
            pointB__dwithin=(OuterRef("pointA"), 10)
        ))
    ).filter(has_value=True)

This fails with "ValueError: This queryset contains a reference to an outer query and may only be used in a subquery"

I dug into things, and while I may not fully understand how queries are processed, I think I've identified the problem in Query.resolve_lookup_value():

def resolve_lookup_value(self, value, can_reuse, allow_joins, simple_col):
    if hasattr(value, 'resolve_expression'):
        kwargs = {'reuse': can_reuse, 'allow_joins': allow_joins}
        if isinstance(value, F):
            kwargs['simple_col'] = simple_col
        value = value.resolve_expression(self, **kwargs)
    elif isinstance(value, (list, tuple)):
        # The items of the iterable may be expressions and therefore need
        # to be resolved independently.
        for sub_value in value:
            if hasattr(sub_value, 'resolve_expression'):
                if isinstance(sub_value, F):
                    sub_value.resolve_expression(
                        self, reuse=can_reuse, allow_joins=allow_joins,
                        simple_col=simple_col,
                    )
                else:
                    sub_value.resolve_expression(self, reuse=can_reuse, allow_joins=allow_joins)
    return value

This resolves the value passed in as the rhs of a filter. For single objects, it calls resolve_expression() and then returns the result. But for multiple objects in a list or tuple, it calls resolve_expression() on each but doesn't return the resolved objects. Rather it returns the original list.

As a consequence, during the call to filter, the passed in OuterRef object doesn't get resolved to a ResolvedOuterRef object, since the __dwithin lookup takes a tuple of (value, distance), triggering that second code path above. Later during the call to annotate, the OuterRef does resolve to a ResolvedOuterRef but when the query is compiled, ResolvedOuterRef.as_sql() is called which raises the error.

I modified resolve_lookup_value() to return the list or tuple of resolved values, and wrote a test for the query above, which I will submit in a PR shortly.

[Andrew Brown]

PR submitted: ​https://github.com/django/django/pull/11634

[Carlton Gibson]

OK, thanks for the report and the test case Andrew.

[Simon Charette]

I left a few comments for improvement but I'm marking as RFC because they are really minor and could be addressed by the committer.

Thanks for the patch and tests Andrew, that was a breeze to review.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 8a281aa7:

Fixed #30687 -- Fixed using of OuterRef() expressions in distance lookups.