Code

Opened 16 months ago

Closed 14 months ago

Last modified 14 months ago

#20157 closed Bug (fixed)

Cannot pickle prefetched queryset on model with callable default datetime.datetime.now

Reported by: thomaspurchas Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Normal Keywords: pickle, queryset
Cc: thomaspurchas, apollo13, bmispelon@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

With the follow models

class Post(models.Model):
    post_date = models.DateTimeField(default=datetime.datetime.now)

class Material(models.Model):
    post = models.ForeignKey(Post, related_name='materials')

When the following is run

post = Post.objects.create()
posts = Post.objects.all()
posts = posts.prefetch_related('materials')
pickle.dumps(posts)

Pickle returns the following error PicklingError: Can't pickle <built-in method now of type object at 0x10782c640>: it's not found as __main__.now.

This appears to be some sort of repeat of issue #13328, and it only affects 1.5. Using bisect I have found the first commit that breaks this 056ace0f395a.

I have also attached a complete traceback.

Attachments (1)

traceback.txt (9.4 KB) - added by thomaspurchas 16 months ago.
Traceback

Download all attachments as: .zip

Change History (14)

Changed 16 months ago by thomaspurchas

Traceback

comment:1 Changed 16 months ago by thomaspurchas

  • Cc thomaspurchas added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 16 months ago by apollo13

I think (!!, not 100% sure, but it seems like a valid start) 69597e5bcc89aadafd1b76abf7efab30ee0b8b1a fixed it in master.

comment:3 Changed 16 months ago by apollo13

  • Cc apollo13 added

comment:4 Changed 16 months ago by apollo13

  • Triage Stage changed from Unreviewed to Accepted

comment:5 Changed 16 months ago by bmispelon

  • Cc bmispelon@… added

comment:6 Changed 16 months ago by thomaspurchas

Yeah sorry, for some reason it never crossed my mind to check the master. The bug appears to be fixed in master.

comment:7 Changed 15 months ago by anonymous

Just to make sure this wasn't missed somewhere, but by default, auth.user uses timezone.now which is exhibiting the same issues (for me) with trying to pickle it, even indirectly with vary headers in the cache.

comment:8 Changed 15 months ago by lukeplant

In theory this could be marked as release blocker because it is a regression. However, the commit which fixed it is a large refactor, which probably would not be a good idea to backport.

Do we have code in Django that pickels auth.User querysets, or any querysets? Is it possible to work around the issue faced by anonymous?

comment:9 Changed 15 months ago by jamesmfriedman@…

For what its worth, I think this is a relatively severe issue if your Django app relies on the low level cache api and you were caching querysets. Is anyone aware of a workaround for the time being? I'm assuming whatever is in master is going out in 1.5.2.

comment:10 Changed 15 months ago by apollo13

No, nothing from master will be in 1.5.2 unless explicitly backported.

comment:11 Changed 14 months ago by Anssi Kääriäinen <akaariai@…>

  • Resolution set to fixed
  • Status changed from new to closed

In bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5:

[1.5.x] Fixed prefetch_related + pickle regressions

There were a couple of regressions related to field pickling. The
regressions were introduced by QuerySet._known_related_objects caching.

The regressions aren't present in master, the fix was likely in
f403653cf146384946e5c879ad2a351768ebc226.

Fixed #20157, fixed #20257. Also made QuerySets with model=None
picklable.

comment:12 Changed 14 months ago by michaelmior

I am seeing this as well without any callable defaults. Take the model below.

class Profile(models.Model):
    user = models.ForeignKey(User, primary_key=True, related_name='profile')

Then the following is not pickleable

User.objects.all().prefetch_related('profile')

I'm going to test the posted fix, but just wondering if that fix will make it into 1.5.2 as this is currently blocking our team from upgrading to Django 1.5. Thanks!

comment:13 Changed 14 months ago by michaelmior

Confirmed that the bug is fixed at bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5. Also realized my bug is actually #20257 and not this.

Last edited 14 months ago by michaelmior (previous) (diff)

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.