Cache pickling exception in 1.8a1 with cross-table filter params

[Mark Tranchant]

In 1.7.4 (and prior versions back to at least 2011), I can run the following transcript in ../manage.py shell. The Contributor model has two ForeignKeys, to MyModel and Task, so I'm looking up all MyModels which have a referring contributor associated with task 4.

>>> from MyProject.models import *
>>> from django.core.cache import cache
>>> cl = MyModel.objects.filter(contributor__task__id=4)
>>> cl
<<< []
>>> type(cl)
<<< django.db.models.query.QuerySet
>>> cache.set('testing', cl)
>>> cache.get('testing')
<<< []

In 1.8a1, the cache set operation generates an exception. See attachment for trace.

PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module

If I try to cache a simple empty QuerySet (MyModel.objects.filter(id=0), for example), it works.

[Mark Tranchant]

Exception trace

[Tim Graham]

Bisected to f233bf47dde1d481108142c8d6b4bb3b3d8c6d08. Adding test and continuing investigation.

[Tim Graham]

The app registry is trying to be pickled and the module that cannot be pickled in my test is from:

<ContentTypesConfig: contenttypes>
{'models_module': <module 'django.contrib.contenttypes.models' from '/home/tim/code/django/django/contrib/contenttypes/models.py'>, 'name': 'django.contrib.contenttypes', 'models': OrderedDict([('contenttype', <class 'django.contrib.contenttypes.models.ContentType'>)]), 'module': <module 'django.contrib.contenttypes' from '/home/tim/code/django/django/contrib/contenttypes/__init__.py'>, 'label': 'contenttypes', 'path': u'/home/tim/code/django/django/contrib/contenttypes'}

I think the solution is to figure out why that's now happening and to prevent it (assuming it can be avoided without loss of functionality).

[Tim Graham]

​PR (might be able to be improved by someone more knowledgeable about pickle). Mark, could you test and verify it also solves your issue?

[Tim Graham]

As Loic mentiond on IRC, a general mechanism for excluding cached_properties from pickling could be useful, but it's a bit out of the scope of solving this issue.

[Mark Tranchant]

I can confirm that adding the __getstate__ function as in the patch to stock 1.8a1 from PyPl fixes my issue, and allows my entire test suite to run without error.

Thanks!

[Tim Graham <timograham@…>]

In f95122e541df5bebb9b5ebb6226b0013e5edc893:

Fixed #24381 -- removed ForeignObjectRel opts and to_opts

These cached properies were causing problems with pickling, and in
addition they were confusingly defined: field.rel.model._meta was
not the same as field.rel.opts.

Instead users should use field.rel.related_model._meta inplace of
field.rel.opts, and field.rel.to._meta in place of field.rel.to_opts.

[Tim Graham <timograham@…>]

In 155a127afbd531180eaa2fd875fe111178cfb64c:

[1.8.x] Fixed #24381 -- removed ForeignObjectRel opts and to_opts

These cached properies were causing problems with pickling, and in
addition they were confusingly defined: field.rel.model._meta was
not the same as field.rel.opts.

Instead users should use field.rel.related_model._meta inplace of
field.rel.opts, and field.rel.to._meta in place of field.rel.to_opts.

Backport of f95122e541df5bebb9b5ebb6226b0013e5edc893 from master

[Claude Paroz <claude@…>]

In 4e7ed8d:

Fixed #24624 -- Replaced obsoleted rel.opts in admindocs view

Thanks Scott Sanders for the report, and Markus Holtermann and
Tim Graham for the reviews. Refs #24381.

[Claude Paroz <claude@…>]

In 774d09a:

[1.8.x] Fixed #24624 -- Replaced obsoleted rel.opts in admindocs view

Thanks Scott Sanders for the report, and Markus Holtermann and
Tim Graham for the reviews. Refs #24381.
Backport of 4e7ed8d0d from master.