Queryset.query pickle fails if fields have lazy translations in field declaration params.

[Jani Tiainen]

If declared model as:

from django.db import models
from django.utils.translation import ugettext_lazy as _

class TestModel(models.Model):
    name = models.CharField(_('name'), max_length=12)

Using query pickle like:

from models import TestModel
import pickle

qs = TestModel.objects.filter(name__contains='foobar')
        
s = pickle.dumps(qs.query)

Fails with exception:

Traceback (most recent call last):
  File "/home/KEYPRO/jtiai/src/django-tests/pickle_error/showcase/tests.py", line 15, in test_pickle_query
    s = pickle.dumps(qs.query)
  File "/usr/lib/python2.6/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.6/pickle.py", line 615, in _batch_appends
    save(x)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'django.utils.functional.__proxy__'>: it's not found as django.utils.functional.__proxy__

This works perfectly with 1.1.1 so it must be a regression.

[Jani Tiainen]

Testcase

[Jani Tiainen]

Main reason was that before multidb there was no pickling support for fields. In multidb pickling appeared but it only took care of lazytranslated error messages. Fields may have at least verbose_name and help_text as lazy translated things.

[Jani Tiainen]

Fix to handle verbose_name and help_text translations while pickling

[Jani Tiainen]

Updated test cases

[Joe Holloway]

I have noticed that this is also an issue if you use ugettext_lazy in the choices argument for a CharField:

my_field = models.CharField(max_length=10, choices=(('1', _('Choice1')), ('2', _('Choice2'))))

I don't think the proposed patch would address this, but I admit that I don't understand why this declarative stuff is even getting pickled -- i.e. whether it's worth patching at this level or there's some overarching issue to address.

[Chris Beaven]

It's a regression. Not sure if it's important enough for 1.2 status, but I'll err on safety and mark as such.

[Jani Tiainen]

Note that pickling queryset.query is documented feature:
"The query attribute is an opaque object. It represents the internals of the query construction and is not part of the public API. However, it is safe (and fully supported) to pickle and unpickle the attribute's contents as described here."

​http://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets

[Chris Beaven]

I've been pondering if it would be possible to make the lazy __proxy__ class picklable. Then we'd solve all of this transparently.

[mrts]

Replying to SmileyChris:

I've been pondering if it would be possible to make the lazy __proxy__ class picklable. Then we'd solve all of this transparently.

I tried to figure that out. Unfortunately, it is not trivial. See ​http://code.google.com/p/modwsgi/wiki/IssuesWithPickleModule for the specifics, but in short -- as far as the name __proxy__ is not available in the module namespace (being dynamically constructed in lazy()), it can't be pickled. It's exactly as the interpreter says: it's not found as django.utils.functional.__proxy__.

Now, via arcane trickery the __proxy__ class can indeed be temporarily inserted into the module namespace in __getstate___ and __setstate__ -- I'll attach a patch for that. Although it works perfectly, it looks really fragile and somewhat insane. Do we need that much magic?

[mrts]

See ​http://dpaste.com/174045/ and ​http://github.com/mrts/django/compare/ticket12769 for tests.

After adding ​http://dpaste.com/174045/ tests pass.

[mrts]

Added tests for the problem.

[mrts]

I've added the trick just for reference and will continue with a saner version in the QuerySet.query layer.

[mrts]

It is actually possible to make __proxy__ pickleable in a cleaner way by overriding __reduce_ex__ (or __reduce__ for that matter, the former is a bit more efficient and the latter is deprecated) as Alex found out.

However, that doesn't play well with delayed loading in utils.translation.delayed_loader. The problems are examined in ​http://botland.oebfare.com/logger/django-dev/2010/3/20/3/ .

But the case is not over yet, let's see how things develop at #12924 before continuing with this.

[Russell Keith-Magee]

(In [12866]) Fixed #12769, #12924 -- Corrected the pickling of curried and lazy objects, which was preventing queries with translated or related fields from being pickled. And lo, Alex Gaynor didst slayeth the dragon.

[Stephan Jaensch]

This is not yet fixed for me. In my session data I have a list of models created by MyModel.objects.create(). This was working fine with 1.1.1 and gave me the error "PicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed" prior to changeset 12866. Post fix, I'm now getting "TypeError: expected string or Unicode object, NoneType found" at "pickled = pickle.dumps(session_dict, pickle.HIGHEST_PROTOCOL)".

A workaround is to do MyModel.objects.get(pk=the_new_pk) immediately after creation and storing this model object in the session.

[Russell Keith-Magee]

The patch contains a regression test to specifically test this case. Without specific details (i.e., not just talking about "MyModel", but actually providing an example of the definition of an unpickleable MyModel), I'm going to close this as fixed. If you can provide *specific* details, please reopen.

[Jacob]

Milestone 1.2 deleted