Opened 8 years ago

Closed 7 years ago

Last modified 5 years ago

#7506 closed (fixed)

Unable to pickle certain QuerySet objects due to use of curry function in RelatedField

Reported by: Michael Malone Owned by: Jacob
Component: Database layer (models, ORM) Version: 1.1-beta
Severity: Keywords: QuerySet pickle curry
Cc: justinarthur@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

QuerySet objects containing Model objects that have a RelatedField attribute are not pickle-able due to the use of the curry function in RelatedField. The Python pickle/cPickle modules can only serialize objects that are defined at the top level of a module. Since the curry function returns a nested function, pickling fails.

In this case, the use of curry to create a partial function seems unnecessary. Equivalent functionality can be achieved by storing the curry parameters in an instance variable. This appears to be the simplest fix to this problem, so it's probably the right one. I've attached a diff for related.py that implements this fix. Alternatively, re-implementing the curry function as a callable class may work, but not without some trickery in getstate to get the callable to pickle.

Attachments (1)

related.diff (2.1 KB) - added by Michael Malone 8 years ago.
Patch to remove curry function from RelatedField init

Download all attachments as: .zip

Change History (22)

Changed 8 years ago by Michael Malone

Attachment: related.diff added

Patch to remove curry function from RelatedField init

comment:1 Changed 8 years ago by Jacob

milestone: 1.0
Needs documentation: unset
Needs tests: unset
Owner: changed from nobody to Jacob
Patch needs improvement: unset
Triage Stage: UnreviewedAccepted

I think this is a duplicate of #7204, but this is actually the root cause. I'm going to keep both open until I verify.

comment:2 Changed 8 years ago by Jacob

Status: newassigned

mmalone: do you have a minimal test case that can replicate this error? Simply calling pickle(qs) for some arbitrary qs doesn't fail; it appears you need to do something else to make that failure happen.

comment:3 Changed 8 years ago by Michael Malone

jacob: I think the model objects inside the QuerySet has to have a RelatedField that you're filtering by. The RelatedField.contribute_to_class() method has to be called. I'm working on narrowing it down at the moment. Here's a dpaste from the console showing the exception, but it's obviously specific to our code base (probably not that helpful, but w/e): http://dpaste.com/57631/

I'll keep working on a minimal test-case to reproduce the error.

comment:4 Changed 8 years ago by Michael Malone

Ok, I've got a working test-case. I've reproduced it using a OneToOneField, not sure if other types of RelatedFields trigger the problem.

Complete models.py:

from django.db import models

class Sandwich(models.Model):
    pass

class Pickle(models.Model):
    sandwich = models.OneToOneField(Sandwich, primary_key=True, editable=False)

Exception in interpreter:

>>> from pickledcurry.app.models import *
>>> from django.db.models.query import QuerySet
>>> qs = QuerySet(Pickle).filter(pk=1)
>>> import pickle
>>> pickle.dumps(qs)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 615, in _batch_appends
    save(x)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 562, in save_tuple
    save(element)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce

    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function _curried at 0x12c42f0>: it's not found as django.utils.functional._curried
>>> 

And here's a little function that will dump a trace of where the PicklingError is being raised, along with the output when run w/ the qs I created in the last example:

def trace_pickle(obj):
  if not hasattr(obj, '__dict__'):
    return
  for k in obj.__dict__:
    try:
      pickle.dumps(obj.__dict__[k])
    except:
      print ("%s -> %s" % (k, obj.__dict__[k]))[:120]
      trace_pickle(obj.__dict__[k])

>>> trace_pickle(qs)
query -> SELECT `app_pickle`.`sandwich_id` FROM `app_pickle` WHERE `app_pickle`.`sandwich_id` = 1 
connection -> <django.db.backends.mysql.base.DatabaseWrapper object at 0x1068750>
connection -> <_mysql.connection open to 'localhost' at a38610>
encoders -> {<type 'int'>: <function Thing2Str at 0x10696b0>, <type 'datetime.timedelta'>: <function DateTimeDelta2liter
string_decoder -> <function string_decoder at 0x12f1330>
unicode_literal -> <function unicode_literal at 0x12f12f0>
where -> (AND: ('app_pickle', 'sandwich_id', <django.db.models.fields.related.OneToOneField object at 0x12b0c30>, 'exact
children -> [('app_pickle', 'sandwich_id', <django.db.models.fields.related.OneToOneField object at 0x12b0c30>, 'exact',

Hope that all makes sense. Let me know if you need any other info.

comment:5 Changed 8 years ago by Jacob

You only put one pickle on your sandwiches? What a sad, sad thing!

(Thanks; I'll play around with this and see if it's what I need.)

comment:6 Changed 8 years ago by anonymous

I've been able to reproduce this with ForeignKey relationships in r7726 of trunk.

comment:7 Changed 8 years ago by anonymous

Cc: justinarthur@… added

comment:8 Changed 8 years ago by anonymous

Here's my question, why did this problem start occurring recently? The curried function was added as part of magic removal in [2381] and merged into trunk in [2809]. Pickling ORM objects with relationships was working just fine for me as of r6626 of trunk.

comment:9 Changed 8 years ago by Michael Malone

When was qs-rf merged? Here's a thread from two months ago where Malcolm acknowledges that QuerySet pickling is broken: http://groups.google.com/group/django-users/browse_thread/thread/32143d024b17dd00/76b336189c3c3757?lnk=st#76b336189c3c3757

The curry function could have been there/can stay. It just can't be pickled. So maybe in r6626 it wasn't being pickled..? One could get fancy and implement getstate/re-init the object when it's retrieved, or something like that, but that seems like more trouble than it's worth. I'm wondering why the curry function is used here to begin with? As far as I can tell, the change I made maintains the same functionality and is as easy/easier to understand than the curry version.

comment:10 Changed 8 years ago by JustinTArthur

Yea, definitely a valid point. Your patch sounds fine. QuerySet Refactor was merged into trunk at r7477.

comment:11 Changed 8 years ago by Malcolm Tredinnick

Resolution: fixed
Status: assignedclosed

Fixed in [7773].

comment:12 Changed 7 years ago by wfagan@…

Has patch: unset
milestone: 1.01.1
Version: SVN1.1-beta-1

I am running Django 1.1 beta and having this same problem I think.

Can't pickle <function _curried at 0x9eb8e9c>: it's not found as django.utils.functional._curried

Here is my query.

query = Model.objects.all().annotate(liked_count=Count('liked_users')).exclude(liked_count=0).order_by('-liked_count')

liked_users is a related name from my UserProfile objects.

im doing ...

pickle.dumps(query)

comment:13 Changed 7 years ago by anonymous

Resolution: fixed
Status: closedreopened

comment:14 Changed 7 years ago by Russell Keith-Magee

Resolution: fixed
Status: reopenedclosed

@wfagan Your problem is slightly different, and is already logged as #10197.

comment:15 Changed 7 years ago by wfagan@…

russellm: thanks

comment:16 Changed 7 years ago by anentropic

Cc: ego@… added
Resolution: fixed
Status: closedreopened

Just upgraded our server to 1.2-beta-1 and started getting this error.
Am trying to cache a simple queryset of a model which has a ForeignKey field... looks like this bug is back?

comment:17 in reply to:  16 Changed 7 years ago by anentropic

Replying to anentropic:

Just upgraded our server to 1.2-beta-1 and started getting this error.
Am trying to cache a simple queryset of a model which has a ForeignKey field... looks like this bug is back?

the qs I'm trying to cache (from a Manager method) is:
self.filter(content_type=ct.pk)

...where content_type field is a ForeignKey.

Hope that helps.

comment:18 Changed 7 years ago by Russell Keith-Magee

Resolution: fixed
Status: reopenedclosed

Please don't reopen an old ticket; This ticket was fixed in [7773], and the fix included a test case, so we know that this specific case can't re-emerge. Instead, open a new ticket, with a specific model and code example that is failing.

comment:19 Changed 7 years ago by anentropic

Ok sorry, have opened a new one at: http://code.djangoproject.com/ticket/13119

comment:20 Changed 7 years ago by anentropic

Cc: ego@… removed

comment:21 Changed 5 years ago by Jacob

milestone: 1.1

Milestone 1.1 deleted

Note: See TracTickets for help on using tickets.
Back to Top