Opened 13 years ago

Closed 12 years ago

Last modified 12 years ago

#16563 closed Bug (fixed)

Error pickling request.user

Reported by: zero.fuxor@… Owned by: nobody
Component: contrib.auth Version: 1.3
Severity: Release blocker Keywords:
Cc: Torsten Bronger, kmike84@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Jannis Leidel)

trying to pickle a request.user in trunk raises: TypeError, can't pickle function objects
Looking it shows that request.user is a SimpleLazyObject and it is a <lambda>, so it can not be pickled.

try:
    import cPickle as pickle
except:
    import pickle


def some_view(request):
    pickle.dumps(request.user) # raise type error
    …

Attachments (2)

ticket_16563.diff (1.6 KB ) - added by Luke Plant 12 years ago.
Patch with test
ticket_16563.2.diff (1.5 KB ) - added by Luke Plant 12 years ago.
Small fix to patch (unneeded import)

Download all attachments as: .zip

Change History (25)

comment:1 by Paul McMillan, 13 years ago

Resolution: wontfix
Status: newclosed

Some things aren't pickleable. It's a known limitation at the Python level, and is unlikely to change. Is Django itself trying to do this somewhere? Is there an extremely good reason this should be possible? I'm closing this as wontfix. Please feel free to reply if you've got good answers to these questions.

comment:2 by Chris Beaven, 13 years ago

milestone: 1.4
Resolution: wontfix
Status: closedreopened
Triage Stage: UnreviewedDesign decision needed

I'm going to reopen for now due to the fact this is a regression in 1.4.

Previously, this was possible:

def my_view(request):
    obj = MyObj(user=request.user, amount=1)
    request.session['unsaved_obj'] = obj
    # ...

Now request.user has been changed to a SimpleLazyObject rather than changing the request class to actually lazily return the true user object, this will choke.
Maybe this is an acceptable regression, but it's something I tripped up on in a project recently.

comment:3 by Russell Keith-Magee, 13 years ago

Severity: NormalRelease blocker

If this is a regression, then it should be tracked as a release blocker.

comment:4 by Jannis Leidel, 13 years ago

Description: modified (diff)

comment:5 by Torsten Bronger, 13 years ago

Cc: Torsten Bronger added

comment:6 by Jacob, 13 years ago

Triage Stage: Design decision neededAccepted

comment:7 by Jacob, 12 years ago

milestone: 1.4

Milestone 1.4 deleted

comment:8 by anonymous, 12 years ago

Same problem with Django 1.4 (trunk), but in Django 1.3 all fine

from django.core.cache import cache
car = Car.objects.get(pk=8809)
car.name = 'test'
car.save()
cache.set('test', request.user, 1)

Raise error:
can't pickle function objects
C:\Python\lib\site-packages\django\core\cache\backends\locmem.py in set

self._set(key, pickle.dumps(value), timeout)

C:\Python\lib\copy_reg.py in _reduce_ex

raise TypeError, "can't pickle %s objects" % base.name

comment:9 by Paul McMillan, 12 years ago

Is this regression related to the change to using pickle.HIGHEST_PROTOCOL throughout Django? The memcached backend used that, but locmem didn't. Does that mean this isn't a regression after all, and depended on the non-canonical behavior of locmem using the lowest pickle protocol?

comment:10 by Chris Beaven, 12 years ago

No, it's due to the fact that a SimpleLazyObject isn't picklable.

comment:11 by Tobias McNulty, 12 years ago

Owner: changed from nobody to Tobias McNulty
Status: reopenednew

comment:12 by Tobias McNulty, 12 years ago

Owner: changed from Tobias McNulty to nobody

comment:13 by Dan Poirier, 12 years ago

I wouldn't call this a regression unless Django promised you could do it in 1.3, at least implicitly. Not every behavior change is a regression.

comment:14 by Dan Poirier, 12 years ago

FYI, the current code that makes request.user a SimpleLazyObject is from changeset [16305], which was a re-working of [16297], to fix #15929.

A comment cleaned up along the way said:

    # If we access request.user, request.session is accessed, which results in 
    # 'Vary: Cookie' being sent in every request that uses this context 
    # processor, which can easily be every request on a site if 
    # TEMPLATE_CONTEXT_PROCESSORS has this context processor added.  This kills 
    # the ability to cache.  So, we carefully ensure these attributes are lazy. 

which seems like a pretty good reason for making request.user lazy.

Would it be possible to fix this instead by fixing the chain of events somewhere else? e.g. should any access of request.session result in setting the Vary: Cookie header?

comment:15 by Tobias McNulty, 12 years ago

As a work around, can't you just pickle a real User model object instead of request.user? E.g., the dumb way to do this might look something like:

def some_view(request):
    pickle.dumps(User.objects.get(pk=request.user.pk))

in reply to:  15 ; comment:16 by Dan Poirier, 12 years ago

Replying to tobias:

As a work around, can't you just pickle a real User model object instead of request.user?

That would still require a code change when upgrading to 1.4, right?

in reply to:  16 comment:17 by Tobias McNulty, 12 years ago

Replying to poirier:

Replying to tobias:

As a work around, can't you just pickle a real User model object instead of request.user?

That would still require a code change when upgrading to 1.4, right?

Yes, I meant it as a complement to your original argument. As you said, being able to pickle request.user directly was never documented, so whether or not it's a release blocking regression is debatable.

in reply to:  14 comment:18 by Carl Meyer, 12 years ago

Replying to poirier:

Would it be possible to fix this instead by fixing the chain of events somewhere else?

No, I don't think so.

e.g. should any access of request.session result in setting the Vary: Cookie header?

Yes, it should. Any access of the session means the response you are generating is almost certainly dependent in some way on values in the session, which means serving that same response as a cached response to other users would be at best wrong, and at worst a security issue. This applies even more strongly, if anything, to accessing request.user in particular. So it's quite important that request.user remain lazy, and that accessing it trigger Vary: Cookie on the response.

comment:19 by Mikhail Korobov, 12 years ago

Cc: kmike84@… added

Are there obstacles for making LazyObject instances picklable? E.g.:

class LazyObject(object):
    # ...

    def __getstate__(self):
        if self._wrapped is None: 
            self._setup()
        return self._wrapped

    def __setstate__(self, state):
        self._wrapped = state

by Luke Plant, 12 years ago

Attachment: ticket_16563.diff added

Patch with test

comment:20 by Luke Plant, 12 years ago

@kmike: That doesn't quite work, but it is close. My attached patch works in some cases, if someone can verify that it fixes the issue with a real User instance in environment showing the bug, I will commit it.

I added the code to SimpleLazyObject not LazyObject, to minimize impact with other subclasses of LazyObject. SimpleLazyObject is not designed to be subclassed really - you are supposed to use it just by passing a callable to the constructor.

There is one potential backwards compatibility concern here: if you had used a SimpleLazyObject with a callable that was itself pickleable, it would previously have been possible to pickle that SimpleLazyObject without the patch. Potentially, the object could have been pickled in the 'lazy' state, without the callable having been called.

However, I've tested this, and it seems that - for whatever reason - calling pickle.dumps causes the the callable to be called. Therefore, the object is not serialised in a 'lazy' state - it is always 'evaluated'. And this is just the same as it is with the patch, except I have made it explicit. The only difference now is that SimpleLazyObject._setupfunc is never available now after unpickling, but that shouldn't be a problem since it is never used.

I'm therefore fairly confident that this patch shouldn't cause other problems.

Last edited 12 years ago by Luke Plant (previous) (diff)

by Luke Plant, 12 years ago

Attachment: ticket_16563.2.diff added

Small fix to patch (unneeded import)

comment:21 by Luke Plant, 12 years ago

OK, this has turned out to be rather tricky.

The previous patch only works for objects that have no state. The basic problem is that due to proxying the __class__ attribute, the pickling machinery is confused (I think), and so it stores a reference to the class of the wrapped object, rather than SimpleLazyObject.

So, we need to define the __reduce__ method for full customization, rather than the simpler __getstate__ method. However, for some reason, the presence of the __class__ attribute proxying causes this to fail - if you define __reduce__ on SimpleLazyObject, it is never called when pickling unless you remove the __class__ hacking.

This takes us back to using __getstate__ (which, for some reason, doesn't suffer from the same problem as __reduce__). But now we have to cooperate with the fact that pickling will skip SimpleLazyObject and store a reference to the wrapped class, and define __getstate__ so that it returns the state from the wrapped object. This appears to work:

    def __getstate__(self):
        if self._wrapped is empty:
            self._setup()
        return self._wrapped.__dict__

The result is that on unpickling, the SimpleLazyObject wrapper disappears entirely, which should be OK.

I still get failure if SimpleLazyObject is wrapping a builtin, and nothing but removing the __class__ proxying appears to be able to fix that. But that bug has always been there, and this change doesn't affect it.

I've confirmed that with this patch you can pickle request.user, and on unpickling get an object that compares equal to the original, so I'm therefore committing my improved patch.

Last edited 12 years ago by Luke Plant (previous) (diff)

comment:22 by Luke Plant, 12 years ago

Resolution: fixed
Status: newclosed

In [17202]:

Fixed #16563 - Error pickling request.user

Thanks to zero.fuxor for the report

Note: See TracTickets for help on using tickets.
Back to Top