Opened 3 months ago

Closed 3 months ago

#30727 closed Bug (fixed)

Pickling a QuerySet evaluates the querysets given to Subquery in annotate.

Reported by: Andrew Brown Owned by: Andrew Brown
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I wrote a test case for tests/queryset_pickle/tests.py modeled after the test from bug #27499 which is very similar.

    def test_pickle_subquery_queryset_not_evaluated(self):
        """
        Verifies that querysets passed into Subquery expressions
        are not evaluated when pickled
        """
        groups = Group.objects.annotate(
            has_event=models.Exists(Event.objects.filter(group_id=models.OuterRef('id')))
        )
        with self.assertNumQueries(0):
            pickle.loads(pickle.dumps(groups.query))

The Subquery class (which is the base for Exists) only stores the underlying query object and throws the QuerySet away (as of this commit, although I don't think it worked before that). So in theory it shouldn't be pickling the QuerySet.

However, the QuerySet is still stored on the instance within the _constructor_args attribute added by the @deconstructible decorator on the BaseExpression base class. So when the inner query object gets pickled, so does the QuerySet, which attempts to evaluate it. In this case, it gets the error "ValueError: This queryset contains a reference to an outer query and may only be used in a subquery." since the inner queryset is being evaluated independently when called from pickle: it calls QuerySet.__getstate__().

I'm not sure what the best solution is here. I made a patch that adds the following override to __getstate__ to the SubQuery class, which fixes the problem and passes all other Django tests on my machine. I'll submit a PR shortly, but welcome any better approaches since I'm not sure what else that may effect.

class Subquery(Expression):
    ...
    def __getstate__(self):
        obj_dict = super().__getstate__()
        obj_dict.pop('_constructor_args', None)
        return obj_dict

Change History (4)

comment:1 Changed 3 months ago by Andrew Brown

Has patch: set

comment:2 Changed 3 months ago by Andrew Brown

Type: UncategorizedBug

comment:3 Changed 3 months ago by felixxm

Owner: changed from nobody to Andrew Brown
Status: newassigned
Summary: Pickling a Query object evaluates querysets in Subquery expressionsPickling a QuerySet evaluates the querysets given to Subquery in annotate.
Triage Stage: UnreviewedAccepted
Version: 2.2master

comment:4 Changed 3 months ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In 691def1:

Fixed #30727 -- Made Subquery pickle without evaluating their QuerySet.

Subquery expression objects, when pickled, were evaluating the QuerySet
objects saved in its _constructor_args attribute.

Note: See TracTickets for help on using tickets.
Back to Top