Code

Opened 4 years ago

Closed 4 years ago

Last modified 3 years ago

#12769 closed (fixed)

Queryset.query pickle fails if fields have lazy translations in field declaration params.

Reported by: jtiai Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: orm pickle query
Cc: jholloway7, sjaensch Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

If declared model as:

from django.db import models
from django.utils.translation import ugettext_lazy as _

class TestModel(models.Model):
    name = models.CharField(_('name'), max_length=12)

Using query pickle like:

from models import TestModel
import pickle

qs = TestModel.objects.filter(name__contains='foobar')
        
s = pickle.dumps(qs.query)

Fails with exception:

Traceback (most recent call last):
  File "/home/KEYPRO/jtiai/src/django-tests/pickle_error/showcase/tests.py", line 15, in test_pickle_query
    s = pickle.dumps(qs.query)
  File "/usr/lib/python2.6/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.6/pickle.py", line 615, in _batch_appends
    save(x)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'django.utils.functional.__proxy__'>: it's not found as django.utils.functional.__proxy__

This works perfectly with 1.1.1 so it must be a regression.

Attachments (5)

pickle_error.tar.gz (2.3 KB) - added by jtiai 4 years ago.
Testcase
bug12769.diff (1.6 KB) - added by jtiai 4 years ago.
Fix to handle verbose_name and help_text translations while pickling
pickle_error_2.tar.gz (2.5 KB) - added by jtiai 4 years ago.
Updated test cases
ticket12769_tests.diff (1.1 KB) - added by mrts 4 years ago.
Added tests for the problem.
ticket12769_insane_but_working_fix.diff (1.1 KB) - added by mrts 4 years ago.

Download all attachments as: .zip

Change History (20)

Changed 4 years ago by jtiai

Testcase

comment:1 Changed 4 years ago by Alex

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 4 years ago by jtiai

Main reason was that before multidb there was no pickling support for fields. In multidb pickling appeared but it only took care of lazytranslated error messages. Fields may have at least verbose_name and help_text as lazy translated things.

Changed 4 years ago by jtiai

Fix to handle verbose_name and help_text translations while pickling

Changed 4 years ago by jtiai

Updated test cases

comment:3 Changed 4 years ago by jtiai

  • Has patch set

comment:4 Changed 4 years ago by jholloway7

  • Cc jholloway7 added

I have noticed that this is also an issue if you use ugettext_lazy in the choices argument for a CharField:

my_field = models.CharField(max_length=10, choices=(('1', _('Choice1')), ('2', _('Choice2'))))

I don't think the proposed patch would address this, but I admit that I don't understand why this declarative stuff is even getting pickled -- i.e. whether it's worth patching at this level or there's some overarching issue to address.

comment:5 Changed 4 years ago by SmileyChris

  • milestone set to 1.2

It's a regression. Not sure if it's important enough for 1.2 status, but I'll err on safety and mark as such.

comment:6 Changed 4 years ago by jtiai

  • Summary changed from Pickle fails if fields have lazy verbose name translations to Queryset.query pickle fails if fields have lazy translations in field declaration params.

Note that pickling queryset.query is documented feature:
"The query attribute is an opaque object. It represents the internals of the query construction and is not part of the public API. However, it is safe (and fully supported) to pickle and unpickle the attribute's contents as described here."

http://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets

comment:7 follow-up: Changed 4 years ago by SmileyChris

I've been pondering if it would be possible to make the lazy __proxy__ class picklable. Then we'd solve all of this transparently.

comment:8 in reply to: ↑ 7 Changed 4 years ago by mrts

Replying to SmileyChris:

I've been pondering if it would be possible to make the lazy __proxy__ class picklable. Then we'd solve all of this transparently.

I tried to figure that out. Unfortunately, it is not trivial. See http://code.google.com/p/modwsgi/wiki/IssuesWithPickleModule for the specifics, but in short -- as far as the name __proxy__ is not available in the module namespace (being dynamically constructed in lazy()), it can't be pickled. It's exactly as the interpreter says: it's not found as django.utils.functional.__proxy__.

Now, via arcane trickery the __proxy__ class can indeed be temporarily inserted into the module namespace in __getstate___ and __setstate__ -- I'll attach a patch for that. Although it works perfectly, it looks really fragile and somewhat insane. Do we need that much magic?

Changed 4 years ago by mrts

Added tests for the problem.

Changed 4 years ago by mrts

comment:10 Changed 4 years ago by mrts

I've added the trick just for reference and will continue with a saner version in the QuerySet.query layer.

comment:11 Changed 4 years ago by mrts

It is actually possible to make __proxy__ pickleable in a cleaner way by overriding __reduce_ex__ (or __reduce__ for that matter, the former is a bit more efficient and the latter is deprecated) as Alex found out.

However, that doesn't play well with delayed loading in utils.translation.delayed_loader. The problems are examined in http://botland.oebfare.com/logger/django-dev/2010/3/20/3/ .

But the case is not over yet, let's see how things develop at #12924 before continuing with this.

comment:12 Changed 4 years ago by russellm

  • Resolution set to fixed
  • Status changed from new to closed

(In [12866]) Fixed #12769, #12924 -- Corrected the pickling of curried and lazy objects, which was preventing queries with translated or related fields from being pickled. And lo, Alex Gaynor didst slayeth the dragon.

comment:13 Changed 4 years ago by sjaensch

  • Cc sjaensch added
  • Has patch unset
  • Resolution fixed deleted
  • Status changed from closed to reopened

This is not yet fixed for me. In my session data I have a list of models created by MyModel.objects.create(). This was working fine with 1.1.1 and gave me the error "PicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed" prior to changeset 12866. Post fix, I'm now getting "TypeError: expected string or Unicode object, NoneType found" at "pickled = pickle.dumps(session_dict, pickle.HIGHEST_PROTOCOL)".

A workaround is to do MyModel.objects.get(pk=the_new_pk) immediately after creation and storing this model object in the session.

comment:14 Changed 4 years ago by russellm

  • Resolution set to fixed
  • Status changed from reopened to closed

The patch contains a regression test to specifically test this case. Without specific details (i.e., not just talking about "MyModel", but actually providing an example of the definition of an unpickleable MyModel), I'm going to close this as fixed. If you can provide *specific* details, please reopen.

comment:15 Changed 3 years ago by jacob

  • milestone 1.2 deleted

Milestone 1.2 deleted

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.