Opened 8 years ago

Closed 7 years ago

Last modified 6 years ago

#12769 closed (fixed)

Queryset.query pickle fails if fields have lazy translations in field declaration params.

Reported by: Jani Tiainen Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: orm pickle query
Cc: Joe Holloway, Stephan Jaensch Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

If declared model as:

from django.db import models
from django.utils.translation import ugettext_lazy as _

class TestModel(models.Model):
    name = models.CharField(_('name'), max_length=12)

Using query pickle like:

from models import TestModel
import pickle

qs = TestModel.objects.filter(name__contains='foobar')
        
s = pickle.dumps(qs.query)

Fails with exception:

Traceback (most recent call last):
  File "/home/KEYPRO/jtiai/src/django-tests/pickle_error/showcase/tests.py", line 15, in test_pickle_query
    s = pickle.dumps(qs.query)
  File "/usr/lib/python2.6/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.6/pickle.py", line 615, in _batch_appends
    save(x)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'django.utils.functional.__proxy__'>: it's not found as django.utils.functional.__proxy__

This works perfectly with 1.1.1 so it must be a regression.

Attachments (5)

pickle_error.tar.gz (2.3 KB) - added by Jani Tiainen 8 years ago.
Testcase
bug12769.diff (1.6 KB) - added by Jani Tiainen 8 years ago.
Fix to handle verbose_name and help_text translations while pickling
pickle_error_2.tar.gz (2.5 KB) - added by Jani Tiainen 8 years ago.
Updated test cases
ticket12769_tests.diff (1.1 KB) - added by mrts 8 years ago.
Added tests for the problem.
ticket12769_insane_but_working_fix.diff (1.1 KB) - added by mrts 8 years ago.

Download all attachments as: .zip

Change History (20)

Changed 8 years ago by Jani Tiainen

Attachment: pickle_error.tar.gz added

Testcase

comment:1 Changed 8 years ago by Alex Gaynor

Triage Stage: UnreviewedAccepted

comment:2 Changed 8 years ago by Jani Tiainen

Main reason was that before multidb there was no pickling support for fields. In multidb pickling appeared but it only took care of lazytranslated error messages. Fields may have at least verbose_name and help_text as lazy translated things.

Changed 8 years ago by Jani Tiainen

Attachment: bug12769.diff added

Fix to handle verbose_name and help_text translations while pickling

Changed 8 years ago by Jani Tiainen

Attachment: pickle_error_2.tar.gz added

Updated test cases

comment:3 Changed 8 years ago by Jani Tiainen

Has patch: set

comment:4 Changed 8 years ago by Joe Holloway

Cc: Joe Holloway added

I have noticed that this is also an issue if you use ugettext_lazy in the choices argument for a CharField:

my_field = models.CharField(max_length=10, choices=(('1', _('Choice1')), ('2', _('Choice2'))))

I don't think the proposed patch would address this, but I admit that I don't understand why this declarative stuff is even getting pickled -- i.e. whether it's worth patching at this level or there's some overarching issue to address.

comment:5 Changed 8 years ago by Chris Beaven

milestone: 1.2

It's a regression. Not sure if it's important enough for 1.2 status, but I'll err on safety and mark as such.

comment:6 Changed 8 years ago by Jani Tiainen

Summary: Pickle fails if fields have lazy verbose name translationsQueryset.query pickle fails if fields have lazy translations in field declaration params.

Note that pickling queryset.query is documented feature:
"The query attribute is an opaque object. It represents the internals of the query construction and is not part of the public API. However, it is safe (and fully supported) to pickle and unpickle the attribute's contents as described here."

http://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets

comment:7 Changed 8 years ago by Chris Beaven

I've been pondering if it would be possible to make the lazy __proxy__ class picklable. Then we'd solve all of this transparently.

comment:8 in reply to:  7 Changed 8 years ago by mrts

Replying to SmileyChris:

I've been pondering if it would be possible to make the lazy __proxy__ class picklable. Then we'd solve all of this transparently.

I tried to figure that out. Unfortunately, it is not trivial. See http://code.google.com/p/modwsgi/wiki/IssuesWithPickleModule for the specifics, but in short -- as far as the name __proxy__ is not available in the module namespace (being dynamically constructed in lazy()), it can't be pickled. It's exactly as the interpreter says: it's not found as django.utils.functional.__proxy__.

Now, via arcane trickery the __proxy__ class can indeed be temporarily inserted into the module namespace in __getstate___ and __setstate__ -- I'll attach a patch for that. Although it works perfectly, it looks really fragile and somewhat insane. Do we need that much magic?

Changed 8 years ago by mrts

Attachment: ticket12769_tests.diff added

Added tests for the problem.

Changed 8 years ago by mrts

comment:10 Changed 8 years ago by mrts

I've added the trick just for reference and will continue with a saner version in the QuerySet.query layer.

comment:11 Changed 8 years ago by mrts

It is actually possible to make __proxy__ pickleable in a cleaner way by overriding __reduce_ex__ (or __reduce__ for that matter, the former is a bit more efficient and the latter is deprecated) as Alex found out.

However, that doesn't play well with delayed loading in utils.translation.delayed_loader. The problems are examined in http://botland.oebfare.com/logger/django-dev/2010/3/20/3/ .

But the case is not over yet, let's see how things develop at #12924 before continuing with this.

comment:12 Changed 7 years ago by Russell Keith-Magee

Resolution: fixed
Status: newclosed

(In [12866]) Fixed #12769, #12924 -- Corrected the pickling of curried and lazy objects, which was preventing queries with translated or related fields from being pickled. And lo, Alex Gaynor didst slayeth the dragon.

comment:13 Changed 7 years ago by Stephan Jaensch

Cc: Stephan Jaensch added
Has patch: unset
Resolution: fixed
Status: closedreopened

This is not yet fixed for me. In my session data I have a list of models created by MyModel.objects.create(). This was working fine with 1.1.1 and gave me the error "PicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed" prior to changeset 12866. Post fix, I'm now getting "TypeError: expected string or Unicode object, NoneType found" at "pickled = pickle.dumps(session_dict, pickle.HIGHEST_PROTOCOL)".

A workaround is to do MyModel.objects.get(pk=the_new_pk) immediately after creation and storing this model object in the session.

comment:14 Changed 7 years ago by Russell Keith-Magee

Resolution: fixed
Status: reopenedclosed

The patch contains a regression test to specifically test this case. Without specific details (i.e., not just talking about "MyModel", but actually providing an example of the definition of an unpickleable MyModel), I'm going to close this as fixed. If you can provide *specific* details, please reopen.

comment:15 Changed 6 years ago by Jacob

milestone: 1.2

Milestone 1.2 deleted

Note: See TracTickets for help on using tickets.
Back to Top