Error pickling request.user

[zero.fuxor@…]

trying to pickle a request.user in trunk raises: TypeError, can't pickle function objects
Looking it shows that request.user is a SimpleLazyObject and it is a <lambda>, so it can not be pickled.

try:
    import cPickle as pickle
except:
    import pickle


def some_view(request):
    pickle.dumps(request.user) # raise type error
    …

[Paul McMillan]

Some things aren't pickleable. It's a known limitation at the Python level, and is unlikely to change. Is Django itself trying to do this somewhere? Is there an extremely good reason this should be possible? I'm closing this as wontfix. Please feel free to reply if you've got good answers to these questions.

[Chris Beaven]

I'm going to reopen for now due to the fact this is a regression in 1.4.

Previously, this was possible:

def my_view(request):
    obj = MyObj(user=request.user, amount=1)
    request.session['unsaved_obj'] = obj
    # ...

Now request.user has been changed to a SimpleLazyObject rather than changing the request class to actually lazily return the true user object, this will choke.
Maybe this is an acceptable regression, but it's something I tripped up on in a project recently.

[Russell Keith-Magee]

If this is a regression, then it should be tracked as a release blocker.

[Jacob]

Milestone 1.4 deleted

[anonymous]

Same problem with Django 1.4 (trunk), but in Django 1.3 all fine

from django.core.cache import cache
car = Car.objects.get(pk=8809)
car.name = 'test'
car.save()
cache.set('test', request.user, 1)

Raise error:
can't pickle function objects
C:\Python\lib\site-packages\django\core\cache\backends\locmem.py in set

self._set(key, pickle.dumps(value), timeout)

C:\Python\lib\copy_reg.py in _reduce_ex

raise TypeError, "can't pickle %s objects" % base.name

[Paul McMillan]

Is this regression related to the change to using pickle.HIGHEST_PROTOCOL throughout Django? The memcached backend used that, but locmem didn't. Does that mean this isn't a regression after all, and depended on the non-canonical behavior of locmem using the lowest pickle protocol?

[Chris Beaven]

No, it's due to the fact that a SimpleLazyObject isn't picklable.

[Dan Poirier]

I wouldn't call this a regression unless Django promised you could do it in 1.3, at least implicitly. Not every behavior change is a regression.

[Dan Poirier]

FYI, the current code that makes request.user a SimpleLazyObject is from changeset [16305], which was a re-working of [16297], to fix #15929.

A comment cleaned up along the way said:

    # If we access request.user, request.session is accessed, which results in 
    # 'Vary: Cookie' being sent in every request that uses this context 
    # processor, which can easily be every request on a site if 
    # TEMPLATE_CONTEXT_PROCESSORS has this context processor added.  This kills 
    # the ability to cache.  So, we carefully ensure these attributes are lazy. 

which seems like a pretty good reason for making request.user lazy.

Would it be possible to fix this instead by fixing the chain of events somewhere else? e.g. should any access of request.session result in setting the Vary: Cookie header?

[Tobias McNulty]

As a work around, can't you just pickle a real User model object instead of request.user? E.g., the dumb way to do this might look something like:

def some_view(request):
    pickle.dumps(User.objects.get(pk=request.user.pk))

[Dan Poirier]

Replying to tobias:

As a work around, can't you just pickle a real User model object instead of request.user?

That would still require a code change when upgrading to 1.4, right?

[Tobias McNulty]

Replying to poirier:

Replying to tobias:

As a work around, can't you just pickle a real User model object instead of request.user?

That would still require a code change when upgrading to 1.4, right?

Yes, I meant it as a complement to your original argument. As you said, being able to pickle request.user directly was never documented, so whether or not it's a release blocking regression is debatable.

[Carl Meyer]

Replying to poirier:

Would it be possible to fix this instead by fixing the chain of events somewhere else?

No, I don't think so.

e.g. should any access of request.session result in setting the Vary: Cookie header?

Yes, it should. Any access of the session means the response you are generating is almost certainly dependent in some way on values in the session, which means serving that same response as a cached response to other users would be at best wrong, and at worst a security issue. This applies even more strongly, if anything, to accessing request.user in particular. So it's quite important that request.user remain lazy, and that accessing it trigger Vary: Cookie on the response.

[Mikhail Korobov]

Are there obstacles for making LazyObject instances picklable? E.g.:

class LazyObject(object):
    # ...

    def __getstate__(self):
        if self._wrapped is None: 
            self._setup()
        return self._wrapped

    def __setstate__(self, state):
        self._wrapped = state

[Luke Plant]

Patch with test

[Luke Plant]

@kmike: That doesn't quite work, but it is close. My attached patch works in some cases, if someone can verify that it fixes the issue with a real User instance in environment showing the bug, I will commit it.

I added the code to SimpleLazyObject not LazyObject, to minimize impact with other subclasses of LazyObject. SimpleLazyObject is not designed to be subclassed really - you are supposed to use it just by passing a callable to the constructor.

There is one potential backwards compatibility concern here: if you had used a SimpleLazyObject with a callable that was itself pickleable, it would previously have been possible to pickle that SimpleLazyObject without the patch. Potentially, the object could have been pickled in the 'lazy' state, without the callable having been called.

However, I've tested this, and it seems that - for whatever reason - calling pickle.dumps causes the the callable to be called. Therefore, the object is not serialised in a 'lazy' state - it is always 'evaluated'. And this is just the same as it currently is, except we have made it explicit. The only difference now is that SimpleLazyObject._setupfunc is never available now after unpickling, but that shouldn't be a problem since it is never used.

I'm therefore fairly confident that this patch shouldn't cause other problems.

[Luke Plant]

Small fix to patch (unneeded import)

[Luke Plant]

OK, this has turned out to be rather tricky.

The previous patch only works for objects that have no state. The basic problem is that due to proxying the __class__ attribute, the pickling machinery is confused (I think), and so it stores a reference to the class of the wrapped object, rather than SimpleLazyObject.

So, we need to define the __reduce__ method for full customization, rather than the simpler __getstate__ method. However, for some reason, the presence of the __class__ attribute proxying causes this to fail - if you define __reduce__ on SimpleLazyObject, it is never called when pickling unless you remove the __class__ hacking.

This takes us back to using __getstate__ (which, for some reason, doesn't suffer from the same problem as __reduce__). But now we have to cooperate with the fact that pickling will skip SimpleLazyObject and store a reference to the wrapped class, and define __getstate__ so that it returns the state from the wrapped object. This appears to work:

    def __getstate__(self):
        if self._wrapped is empty:
            self._setup()
        return self._wrapped.__dict__

The result is that on unpickling, the SimpleLazyObject wrapper disappears entirely, which should be OK.

I still get failure if SimpleLazyObject is wrapping a builtin, and nothing but removing the __class__ proxying appears to be able to fix that. But that bug has always been there, and this change doesn't affect it.

I've confirmed that with this patch you can pickle request.user, and on unpickling get an object that compares equal to the original, so I'm therefore committing my improved patch.

[Luke Plant]

In [17202]:

Fixed #16563 - Error pickling request.user

Thanks to zero.fuxor for the report

[Jeremy Dunck]

See related ticket: https://code.djangoproject.com/ticket/18538