Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#20157 closed Bug (fixed)

Cannot pickle prefetched queryset on model with callable default datetime.datetime.now

Reported by: thomaspurchas Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Normal Keywords: pickle, queryset
Cc: thomaspurchas, Florian Apolloner, bmispelon@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

With the follow models

class Post(models.Model):
    post_date = models.DateTimeField(default=datetime.datetime.now)

class Material(models.Model):
    post = models.ForeignKey(Post, related_name='materials')

When the following is run

post = Post.objects.create()
posts = Post.objects.all()
posts = posts.prefetch_related('materials')
pickle.dumps(posts)

Pickle returns the following error PicklingError: Can't pickle <built-in method now of type object at 0x10782c640>: it's not found as __main__.now.

This appears to be some sort of repeat of issue #13328, and it only affects 1.5. Using bisect I have found the first commit that breaks this 056ace0f395a.

I have also attached a complete traceback.

Attachments (1)

traceback.txt (9.4 KB ) - added by thomaspurchas 11 years ago.
Traceback

Download all attachments as: .zip

Change History (14)

by thomaspurchas, 11 years ago

Attachment: traceback.txt added

Traceback

comment:1 by thomaspurchas, 11 years ago

Cc: thomaspurchas added

comment:2 by Florian Apolloner, 11 years ago

I think (!!, not 100% sure, but it seems like a valid start) 69597e5bcc89aadafd1b76abf7efab30ee0b8b1a fixed it in master.

comment:3 by Florian Apolloner, 11 years ago

Cc: Florian Apolloner added

comment:4 by Florian Apolloner, 11 years ago

Triage Stage: UnreviewedAccepted

comment:5 by Baptiste Mispelon, 11 years ago

Cc: bmispelon@… added

comment:6 by thomaspurchas, 11 years ago

Yeah sorry, for some reason it never crossed my mind to check the master. The bug appears to be fixed in master.

comment:7 by anonymous, 11 years ago

Just to make sure this wasn't missed somewhere, but by default, auth.user uses timezone.now which is exhibiting the same issues (for me) with trying to pickle it, even indirectly with vary headers in the cache.

comment:8 by Luke Plant, 11 years ago

In theory this could be marked as release blocker because it is a regression. However, the commit which fixed it is a large refactor, which probably would not be a good idea to backport.

Do we have code in Django that pickels auth.User querysets, or any querysets? Is it possible to work around the issue faced by anonymous?

comment:9 by jamesmfriedman@…, 11 years ago

For what its worth, I think this is a relatively severe issue if your Django app relies on the low level cache api and you were caching querysets. Is anyone aware of a workaround for the time being? I'm assuming whatever is in master is going out in 1.5.2.

comment:10 by Florian Apolloner, 11 years ago

No, nothing from master will be in 1.5.2 unless explicitly backported.

comment:11 by Anssi Kääriäinen <akaariai@…>, 11 years ago

Resolution: fixed
Status: newclosed

In bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5:

[1.5.x] Fixed prefetch_related + pickle regressions

There were a couple of regressions related to field pickling. The
regressions were introduced by QuerySet._known_related_objects caching.

The regressions aren't present in master, the fix was likely in
f403653cf146384946e5c879ad2a351768ebc226.

Fixed #20157, fixed #20257. Also made QuerySets with model=None
picklable.

comment:12 by Michael Mior, 11 years ago

I am seeing this as well without any callable defaults. Take the model below.

class Profile(models.Model):
    user = models.ForeignKey(User, primary_key=True, related_name='profile')

Then the following is not pickleable

User.objects.all().prefetch_related('profile')

I'm going to test the posted fix, but just wondering if that fix will make it into 1.5.2 as this is currently blocking our team from upgrading to Django 1.5. Thanks!

comment:13 by Michael Mior, 11 years ago

Confirmed that the bug is fixed at bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5. Also realized my bug is actually #20257 and not this.

Last edited 11 years ago by Michael Mior (previous) (diff)
Note: See TracTickets for help on using tickets.
Back to Top