Unable to pickle certain QuerySet objects due to use of curry function in RelatedField

Patch to remove curry function from RelatedField? init

I think this is a duplicate of #7204, but this is actually the root cause. I'm going to keep both open until I verify.

mmalone: do you have a minimal test case that can replicate this error? Simply calling pickle(qs) for some arbitrary qs doesn't fail; it appears you need to do something else to make that failure happen.

jacob: I think the model objects inside the QuerySet? has to have a RelatedField? that you're filtering by. The RelatedField?.contribute_to_class() method has to be called. I'm working on narrowing it down at the moment. Here's a dpaste from the console showing the exception, but it's obviously specific to our code base (probably not that helpful, but w/e): http://dpaste.com/57631/

I'll keep working on a minimal test-case to reproduce the error.

Ok, I've got a working test-case. I've reproduced it using a OneToOneField?, not sure if other types of RelatedFields? trigger the problem.

Complete models.py:

from django.db import models

class Sandwich(models.Model):
    pass

class Pickle(models.Model):
    sandwich = models.OneToOneField(Sandwich, primary_key=True, editable=False)

Exception in interpreter:

>>> from pickledcurry.app.models import *
>>> from django.db.models.query import QuerySet
>>> qs = QuerySet(Pickle).filter(pk=1)
>>> import pickle
>>> pickle.dumps(qs)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 615, in _batch_appends
    save(x)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 562, in save_tuple
    save(element)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce

    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function _curried at 0x12c42f0>: it's not found as django.utils.functional._curried
>>> 

And here's a little function that will dump a trace of where the PicklingError? is being raised, along with the output when run w/ the qs I created in the last example:

def trace_pickle(obj):
  if not hasattr(obj, '__dict__'):
    return
  for k in obj.__dict__:
    try:
      pickle.dumps(obj.__dict__[k])
    except:
      print ("%s -> %s" % (k, obj.__dict__[k]))[:120]
      trace_pickle(obj.__dict__[k])

>>> trace_pickle(qs)
query -> SELECT `app_pickle`.`sandwich_id` FROM `app_pickle` WHERE `app_pickle`.`sandwich_id` = 1 
connection -> <django.db.backends.mysql.base.DatabaseWrapper object at 0x1068750>
connection -> <_mysql.connection open to 'localhost' at a38610>
encoders -> {<type 'int'>: <function Thing2Str at 0x10696b0>, <type 'datetime.timedelta'>: <function DateTimeDelta2liter
string_decoder -> <function string_decoder at 0x12f1330>
unicode_literal -> <function unicode_literal at 0x12f12f0>
where -> (AND: ('app_pickle', 'sandwich_id', <django.db.models.fields.related.OneToOneField object at 0x12b0c30>, 'exact
children -> [('app_pickle', 'sandwich_id', <django.db.models.fields.related.OneToOneField object at 0x12b0c30>, 'exact',

Hope that all makes sense. Let me know if you need any other info.

You only put one pickle on your sandwiches? What a sad, sad thing!

(Thanks; I'll play around with this and see if it's what I need.)

I've been able to reproduce this with ForeignKey relationships in r7726 of trunk.

Here's my question, why did this problem start occurring recently? The curried function was added as part of magic removal in [2381] and merged into trunk in [2809]. Pickling ORM objects with relationships was working just fine for me as of r6626 of trunk.

When was qs-rf merged? Here's a thread from two months ago where Malcolm acknowledges that QuerySet? pickling is broken: http://groups.google.com/group/django-users/browse_thread/thread/32143d024b17dd00/76b336189c3c3757?lnk=st#76b336189c3c3757

The curry function could have been there/can stay. It just can't be pickled. So maybe in r6626 it wasn't being pickled..? One could get fancy and implement getstate/re-init the object when it's retrieved, or something like that, but that seems like more trouble than it's worth. I'm wondering why the curry function is used here to begin with? As far as I can tell, the change I made maintains the same functionality and is as easy/easier to understand than the curry version.

Yea, definitely a valid point. Your patch sounds fine. QuerySet? Refactor was merged into trunk at r7477.

Fixed in [7773].