Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#20157 closed Bug (fixed)

Cannot pickle prefetched queryset on model with callable default

Reported by: thomaspurchas Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Normal Keywords: pickle, queryset
Cc: thomaspurchas, apollo13, bmispelon@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


With the follow models

class Post(models.Model):
    post_date = models.DateTimeField(

class Material(models.Model):
    post = models.ForeignKey(Post, related_name='materials')

When the following is run

post = Post.objects.create()
posts = Post.objects.all()
posts = posts.prefetch_related('materials')

Pickle returns the following error PicklingError: Can't pickle <built-in method now of type object at 0x10782c640>: it's not found as

This appears to be some sort of repeat of issue #13328, and it only affects 1.5. Using bisect I have found the first commit that breaks this 056ace0f395a.

I have also attached a complete traceback.

Attachments (1)

traceback.txt (9.4 KB) - added by thomaspurchas 2 years ago.

Download all attachments as: .zip

Change History (14)

Changed 2 years ago by thomaspurchas


comment:1 Changed 2 years ago by thomaspurchas

  • Cc thomaspurchas added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 2 years ago by apollo13

I think (!!, not 100% sure, but it seems like a valid start) 69597e5bcc89aadafd1b76abf7efab30ee0b8b1a fixed it in master.

comment:3 Changed 2 years ago by apollo13

  • Cc apollo13 added

comment:4 Changed 2 years ago by apollo13

  • Triage Stage changed from Unreviewed to Accepted

comment:5 Changed 2 years ago by bmispelon

  • Cc bmispelon@… added

comment:6 Changed 2 years ago by thomaspurchas

Yeah sorry, for some reason it never crossed my mind to check the master. The bug appears to be fixed in master.

comment:7 Changed 2 years ago by anonymous

Just to make sure this wasn't missed somewhere, but by default, auth.user uses which is exhibiting the same issues (for me) with trying to pickle it, even indirectly with vary headers in the cache.

comment:8 Changed 2 years ago by lukeplant

In theory this could be marked as release blocker because it is a regression. However, the commit which fixed it is a large refactor, which probably would not be a good idea to backport.

Do we have code in Django that pickels auth.User querysets, or any querysets? Is it possible to work around the issue faced by anonymous?

comment:9 Changed 2 years ago by jamesmfriedman@…

For what its worth, I think this is a relatively severe issue if your Django app relies on the low level cache api and you were caching querysets. Is anyone aware of a workaround for the time being? I'm assuming whatever is in master is going out in 1.5.2.

comment:10 Changed 2 years ago by apollo13

No, nothing from master will be in 1.5.2 unless explicitly backported.

comment:11 Changed 2 years ago by Anssi Kääriäinen <akaariai@…>

  • Resolution set to fixed
  • Status changed from new to closed

In bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5:

[1.5.x] Fixed prefetch_related + pickle regressions

There were a couple of regressions related to field pickling. The
regressions were introduced by QuerySet._known_related_objects caching.

The regressions aren't present in master, the fix was likely in

Fixed #20157, fixed #20257. Also made QuerySets with model=None

comment:12 Changed 2 years ago by michaelmior

I am seeing this as well without any callable defaults. Take the model below.

class Profile(models.Model):
    user = models.ForeignKey(User, primary_key=True, related_name='profile')

Then the following is not pickleable


I'm going to test the posted fix, but just wondering if that fix will make it into 1.5.2 as this is currently blocking our team from upgrading to Django 1.5. Thanks!

comment:13 Changed 2 years ago by michaelmior

Confirmed that the bug is fixed at bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5.

Version 0, edited 2 years ago by michaelmior (next)
Note: See TracTickets for help on using tickets.
Back to Top