Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#24381 closed Bug (fixed)

Cache pickling exception in 1.8a1 with cross-table filter params

Reported by: Mark Tranchant Owned by: Tim Graham
Component: Core (Cache system) Version: 1.8alpha1
Severity: Release blocker Keywords: cache pickle empty queryset
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Mark Tranchant)

In 1.7.4 (and prior versions back to at least 2011), I can run the following transcript in ../manage.py shell. The Contributor model has two ForeignKeys, to MyModel and Task, so I'm looking up all MyModels which have a referring contributor associated with task 4.

>>> from MyProject.models import *
>>> from django.core.cache import cache
>>> cl = MyModel.objects.filter(contributor__task__id=4)
>>> cl
<<< []
>>> type(cl)
<<< django.db.models.query.QuerySet
>>> cache.set('testing', cl)
>>> cache.get('testing')
<<< []

In 1.8a1, the cache set operation generates an exception. See attachment for trace.

PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module

If I try to cache a simple empty QuerySet (MyModel.objects.filter(id=0), for example), it works.

Attachments (2)

pickling-error.txt (13.7 KB) - added by Mark Tranchant 7 years ago.
Exception trace
24381-test.diff (577 bytes) - added by Tim Graham 7 years ago.

Download all attachments as: .zip

Change History (15)

Changed 7 years ago by Mark Tranchant

Attachment: pickling-error.txt added

Exception trace

comment:1 Changed 7 years ago by Mark Tranchant

Summary: Cache pickling issue in 1.8a1Cache pickling exception in 1.8a1

comment:2 Changed 7 years ago by Mark Tranchant

Description: modified (diff)

comment:3 Changed 7 years ago by Mark Tranchant

Description: modified (diff)
Summary: Cache pickling exception in 1.8a1Cache pickling exception in 1.8a1 with cross-table filter params

comment:4 Changed 7 years ago by Mark Tranchant

Description: modified (diff)

comment:5 Changed 7 years ago by Tim Graham

Owner: changed from nobody to Tim Graham
Status: newassigned
Triage Stage: UnreviewedAccepted

Bisected to f233bf47dde1d481108142c8d6b4bb3b3d8c6d08. Adding test and continuing investigation.

Changed 7 years ago by Tim Graham

Attachment: 24381-test.diff added

comment:6 Changed 7 years ago by Tim Graham

The app registry is trying to be pickled and the module that cannot be pickled in my test is from:

<ContentTypesConfig: contenttypes>
{'models_module': <module 'django.contrib.contenttypes.models' from '/home/tim/code/django/django/contrib/contenttypes/models.py'>, 'name': 'django.contrib.contenttypes', 'models': OrderedDict([('contenttype', <class 'django.contrib.contenttypes.models.ContentType'>)]), 'module': <module 'django.contrib.contenttypes' from '/home/tim/code/django/django/contrib/contenttypes/__init__.py'>, 'label': 'contenttypes', 'path': u'/home/tim/code/django/django/contrib/contenttypes'}

I think the solution is to figure out why that's now happening and to prevent it (assuming it can be avoided without loss of functionality).

comment:7 Changed 7 years ago by Tim Graham

Has patch: set

PR (might be able to be improved by someone more knowledgeable about pickle). Mark, could you test and verify it also solves your issue?

comment:8 Changed 7 years ago by Tim Graham

As Loic mentiond on IRC, a general mechanism for excluding cached_properties from pickling could be useful, but it's a bit out of the scope of solving this issue.

comment:9 Changed 7 years ago by Mark Tranchant

I can confirm that adding the __getstate__ function as in the patch to stock 1.8a1 from PyPl fixes my issue, and allows my entire test suite to run without error.

Thanks!

comment:10 Changed 7 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In f95122e541df5bebb9b5ebb6226b0013e5edc893:

Fixed #24381 -- removed ForeignObjectRel opts and to_opts

These cached properies were causing problems with pickling, and in
addition they were confusingly defined: field.rel.model._meta was
not the same as field.rel.opts.

Instead users should use field.rel.related_model._meta inplace of
field.rel.opts, and field.rel.to._meta in place of field.rel.to_opts.

comment:11 Changed 7 years ago by Tim Graham <timograham@…>

In 155a127afbd531180eaa2fd875fe111178cfb64c:

[1.8.x] Fixed #24381 -- removed ForeignObjectRel opts and to_opts

These cached properies were causing problems with pickling, and in
addition they were confusingly defined: field.rel.model._meta was
not the same as field.rel.opts.

Instead users should use field.rel.related_model._meta inplace of
field.rel.opts, and field.rel.to._meta in place of field.rel.to_opts.

Backport of f95122e541df5bebb9b5ebb6226b0013e5edc893 from master

comment:12 Changed 7 years ago by Claude Paroz <claude@…>

In 4e7ed8d:

Fixed #24624 -- Replaced obsoleted rel.opts in admindocs view

Thanks Scott Sanders for the report, and Markus Holtermann and
Tim Graham for the reviews. Refs #24381.

comment:13 Changed 7 years ago by Claude Paroz <claude@…>

In 774d09a:

[1.8.x] Fixed #24624 -- Replaced obsoleted rel.opts in admindocs view

Thanks Scott Sanders for the report, and Markus Holtermann and
Tim Graham for the reviews. Refs #24381.
Backport of 4e7ed8d0d from master.

Note: See TracTickets for help on using tickets.
Back to Top