QuerySet that prefetches related object with a ManyToMany field cannot be pickled.

[bryced]

After upgrading from 1.4 to 1.5, exceptions were thrown when trying to pickle certain querysets. This means that the caching framework doesn't work for these querysets.

In 1.4 the following code runs fine. In 1.5 this error occurs:

PicklingError: Can't pickle <class 'people.models.SocialProfile_friends'>: it's not found as people.models.SocialProfile_friends

models.py

from django.db import models


class Person(models.Model):
    name = models.CharField(max_length=200)


class SocialProfile(models.Model):
    person = models.ForeignKey(Person)
    friends = models.ManyToManyField('self')

tests.py

from django.test import TestCase
from people.models import Person
import pickle


class SimpleTest(TestCase):

    def test_pickle_failure(self):
        bob = Person(name="Bob")
        bob.save()

        people = Person.objects.all().prefetch_related('socialprofile_set')
        pickle.dumps(people)

[Baptiste Mispelon]

Hi,

Thanks for the detailed report.

I can reproduce the issue in the stable/1.5.x branch but it appears to have been fixed in master.

The testcase works on the stable/1.4.x branch so it is a regression.

Using git bisect, I found that the problem was introduced by commit 056ace0f395a58eeac03da9f9ee7e3872e1e407b.

[Anssi Kääriäinen]

The problem is that fields for automatic models (for example the intermediary m2m table) can't be pickled in 1.5. In master this is fixed by introducing Field.reduce. In 1.4 the field wasn't used directly, only the field.name was used so there was no problems. So, to fix this, one of the following three things needs to be done:

• alter QuerySet.__getstate__ and __setstate__ to do something similar that sql.query:Query's state methods do. That is, store the field's name in getstate and restore back the real field instance in setstate.
• remove direct storage of the field in QuerySet
• backpatch Field.reduce changes from master

I think the Field.reduce changes are too risky to backpatch, and I assume there was a good reason to store the field directly in QuerySet instead of field.name. So, changes to getstate and setstate seem like the best choice to me.

[jamesmfriedman@…]

This bug is duplicate of #20157. I've been tracking on it since it is causing me some serious headaches in production since upgrading to 1.5. I do hope for a patch in 1.5.2.

[Anssi Kääriäinen <akaariai@…>]

In bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5:

[1.5.x] Fixed prefetch_related + pickle regressions

There were a couple of regressions related to field pickling. The
regressions were introduced by QuerySet._known_related_objects caching.

The regressions aren't present in master, the fix was likely in
f403653cf146384946e5c879ad2a351768ebc226.

Fixed #20157, fixed #20257. Also made QuerySets with model=None
picklable.

[Michael Mior]

This is currently blocking our team from upgrading to Django 1.5. Just wondering if this will be included in the 1.5.2 release? Thanks for the fix!

[Tim Graham]

@michaelmior, Yes it will be. You can see the commit above is prefixed with [1.5.x] indicating it was committed to the stable/1.5.x branch.

[Michael Mior]

Thanks @timo! I noticed that after. I apologize if it's bad form to ask, but any ETA on the 1.5.2 release? I'm happy to pitch in if there are release blockers that need to be resolved.