Opened 3 weeks ago

Last modified 10 days ago

#37430 assigned Cleanup/optimization

Make database selection for subqueries consistent and document its behavior

Reported by: Lev Zlobin Owned by: Lev Zlobin
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: documentation queryset multiple-databases subquery routing
Cc: Simon Charette Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

  • Without an explicitly selected database, the subquery uses the outer query's connection rather than being routed independently.
  • Explicitly selecting a different database for the inner queryset raises ValueError.
  • Evaluating the inner queryset with list() allows values to be retrieved from another database.

Add examples to the multiple-databases guide and cross-references from the using() and in lookup documentation.

This documentation improvement was split out from https://code.djangoproject.com/ticket/37427 following review on github ​https://github.com/django/django/pull/22154.

Change History (1)

comment:1 by Sarah Boyce, 10 days ago

Cc: Simon Charette added
Component: Documentation → Database layer (models, ORM)
Easy pickings: unset
Summary: Clarify database selection when a queryset is used in an {{{__in}}} lookup: → Make database selection for subqueries consistent and document its behavior
Triage Stage: Unreviewed → Accepted

From what I can see, the outer query’s database connection is also used for Subquery() and Exists() but unlike the in lookup, these silently ignore an explicit database selection on the inner queryset. See below:

  • tests/multiple_database/tests.py

    a b from django.contrib.auth.models import User  
    88from django.contrib.contenttypes.models import ContentType
    99from django.core import management
    1010from django.db import DEFAULT_DB_ALIAS, router, transaction
    11 from django.db.models import signals
     11from django.db.models import Exists, Subquery, signals
    1212from django.db.utils import ConnectionRouter
    1313from django.test import SimpleTestCase, TestCase, override_settings, skipIfDBFeature
    1414
    … … class RouterTestCase(TestCase):  
    19631963        # If you evaluate the query, it should work, running on 'other'
    19641964        self.assertEqual(list(qs.values_list("title", flat=True)), ["Dive into Python"])
    19651965
     1966    def test_subquery_explicit_database_selection(self):
     1967        Person.objects.using("default").create(name="Default person")
     1968        Person.objects.using("other").create(name="Other person")
     1969        book = Book.objects.using("other").create(
     1970            title="Test book",
     1971            published=datetime.date(2026, 1, 1),
     1972        )
     1973
     1974        inner = Person.objects.using("default").order_by("name")
     1975        outer = Book.objects.using("other").filter(pk=book.pk)
     1976
     1977        # Should Subquery() and Exists() raise ValueError for conflicting
     1978        # database selections, as the __in lookup does?
     1979        with self.assertNumQueries(0, using="default"):
     1980            with self.assertNumQueries(1, using="other"):
     1981                result = outer.annotate(
     1982                    person_name=Subquery(inner.values("name")[:1]),
     1983                    has_default_person=Exists(inner.filter(name="Default person")),
     1984                ).get()
     1985
     1986        self.assertEqual(result.person_name, "Other person")
     1987        self.assertIs(result.has_default_person, False)
     1988
     1989        with self.assertRaisesMessage(
     1990            ValueError,
     1991            "Subqueries aren't allowed across different databases.",
     1992        ):
     1993            list(outer.filter(title__in=inner.values("name")))
     1994
    19661995    def test_deferred_models(self):

Perhaps we should document the general database-selection rule in the multiple-databases guide, with cross-references from in, Subquery(), and Exists().
It might also be worth considering whether conflicting explicit database selections should consistently raise an error like what was introduced in #25811

Note: See TracTickets for help on using tickets.
Back to Top