Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#20257 closed Bug (fixed)

QuerySet that prefetches related object with a ManyToMany field cannot be pickled.

Reported by: bryced Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Release blocker Keywords: ORM, pickle, cache
Cc: bmispelon@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


After upgrading from 1.4 to 1.5, exceptions were thrown when trying to pickle certain querysets. This means that the caching framework doesn't work for these querysets.

In 1.4 the following code runs fine. In 1.5 this error occurs:

PicklingError: Can't pickle <class 'people.models.SocialProfile_friends'>: it's not found as people.models.SocialProfile_friends

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=200)

class SocialProfile(models.Model):
    person = models.ForeignKey(Person)
    friends = models.ManyToManyField('self')

from django.test import TestCase
from people.models import Person
import pickle

class SimpleTest(TestCase):

    def test_pickle_failure(self):
        bob = Person(name="Bob")

        people = Person.objects.all().prefetch_related('socialprofile_set')

Change History (7)

comment:1 Changed 3 years ago by bmispelon

  • Cc bmispelon@… added
  • Component changed from Core (Cache system) to Database layer (models, ORM)
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted


Thanks for the detailed report.

I can reproduce the issue in the stable/1.5.x branch but it appears to have been fixed in master.

The testcase works on the stable/1.4.x branch so it is a regression.

Using git bisect, I found that the problem was introduced by commit 056ace0f395a58eeac03da9f9ee7e3872e1e407b.

comment:2 Changed 3 years ago by akaariai

The problem is that fields for automatic models (for example the intermediary m2m table) can't be pickled in 1.5. In master this is fixed by introducing Field.reduce. In 1.4 the field wasn't used directly, only the was used so there was no problems. So, to fix this, one of the following three things needs to be done:

  • alter QuerySet.__getstate__ and __setstate__ to do something similar that sql.query:Query's state methods do. That is, store the field's name in getstate and restore back the real field instance in setstate.
  • remove direct storage of the field in QuerySet
  • backpatch Field.reduce changes from master

I think the Field.reduce changes are too risky to backpatch, and I assume there was a good reason to store the field directly in QuerySet instead of So, changes to getstate and setstate seem like the best choice to me.

comment:3 Changed 3 years ago by jamesmfriedman@…

This bug is duplicate of #20157. I've been tracking on it since it is causing me some serious headaches in production since upgrading to 1.5. I do hope for a patch in 1.5.2.

comment:4 Changed 3 years ago by Anssi Kääriäinen <akaariai@…>

  • Resolution set to fixed
  • Status changed from new to closed

In bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5:

[1.5.x] Fixed prefetch_related + pickle regressions

There were a couple of regressions related to field pickling. The
regressions were introduced by QuerySet._known_related_objects caching.

The regressions aren't present in master, the fix was likely in

Fixed #20157, fixed #20257. Also made QuerySets with model=None

comment:5 Changed 2 years ago by michaelmior

This is currently blocking our team from upgrading to Django 1.5. Just wondering if this will be included in the 1.5.2 release? Thanks for the fix!

comment:6 Changed 2 years ago by timo

@michaelmior, Yes it will be. You can see the commit above is prefixed with [1.5.x] indicating it was committed to the stable/1.5.x branch.

comment:7 Changed 2 years ago by michaelmior

Thanks @timo! I noticed that after. I apologize if it's bad form to ask, but any ETA on the 1.5.2 release? I'm happy to pitch in if there are release blockers that need to be resolved.

Note: See TracTickets for help on using tickets.
Back to Top