Cannot pickle prefetched queryset on model with callable default datetime.datetime.now

[thomaspurchas]

With the follow models

class Post(models.Model):
    post_date = models.DateTimeField(default=datetime.datetime.now)

class Material(models.Model):
    post = models.ForeignKey(Post, related_name='materials')

When the following is run

post = Post.objects.create()
posts = Post.objects.all()
posts = posts.prefetch_related('materials')
pickle.dumps(posts)

Pickle returns the following error PicklingError: Can't pickle <built-in method now of type object at 0x10782c640>: it's not found as __main__.now.

This appears to be some sort of repeat of issue #13328, and it only affects 1.5. Using bisect I have found the first commit that breaks this ​056ace0f395a.

I have also attached a complete traceback.

[thomaspurchas]

Traceback

[Florian Apolloner]

I think (!!, not 100% sure, but it seems like a valid start) 69597e5bcc89aadafd1b76abf7efab30ee0b8b1a fixed it in master.

[thomaspurchas]

Yeah sorry, for some reason it never crossed my mind to check the master. The bug appears to be fixed in master.

[anonymous]

Just to make sure this wasn't missed somewhere, but by default, auth.user uses timezone.now which is exhibiting the same issues (for me) with trying to pickle it, even indirectly with vary headers in the cache.

[Luke Plant]

In theory this could be marked as release blocker because it is a regression. However, the commit which fixed it is a large refactor, which probably would not be a good idea to backport.

Do we have code in Django that pickels auth.User querysets, or any querysets? Is it possible to work around the issue faced by anonymous?

[jamesmfriedman@…]

For what its worth, I think this is a relatively severe issue if your Django app relies on the low level cache api and you were caching querysets. Is anyone aware of a workaround for the time being? I'm assuming whatever is in master is going out in 1.5.2.

[Florian Apolloner]

No, nothing from master will be in 1.5.2 unless explicitly backported.

[Anssi Kääriäinen <akaariai@…>]

In bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5:

[1.5.x] Fixed prefetch_related + pickle regressions

There were a couple of regressions related to field pickling. The
regressions were introduced by QuerySet._known_related_objects caching.

The regressions aren't present in master, the fix was likely in
f403653cf146384946e5c879ad2a351768ebc226.

Fixed #20157, fixed #20257. Also made QuerySets with model=None
picklable.

[Michael Mior]

I am seeing this as well without any callable defaults. Take the model below.

class Profile(models.Model):
    user = models.ForeignKey(User, primary_key=True, related_name='profile')

Then the following is not pickleable

User.objects.all().prefetch_related('profile')

I'm going to test the posted fix, but just wondering if that fix will make it into 1.5.2 as this is currently blocking our team from upgrading to Django 1.5. Thanks!

[Michael Mior]

Confirmed that the bug is fixed at bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5. Also realized my bug is actually #20257 and not this.