to_python being called with unicode text breaks Unpickler

[Mark Jones]

I have a database column that contains a pickle. When the model tries to access it as a custom field, I run into this problem:

>>> QuizQuestion.objects.get(pk=1)
class= <type 'unicode'>
(I1
I2
I3
I4
tp0
.

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "c:\Python25\Lib\site-packages\django\db\models\manager.py", line 93, in get
    return self.get_query_set().get(*args, **kwargs)
  File "c:\Python25\Lib\site-packages\django\db\models\query.py", line 304, in get
    num = len(clone)
  File "c:\Python25\Lib\site-packages\django\db\models\query.py", line 160, in __len__
    self._result_cache = list(self.iterator())
  File "c:\Python25\Lib\site-packages\django\db\models\query.py", line 280, in iterator
    obj = self.model(*row[index_start:])
  File "c:\Python25\Lib\site-packages\django\db\models\base.py", line 213, in __init__
    setattr(self, field.attname, val)
  File "c:\Python25\Lib\site-packages\django\db\models\fields\subclassing.py", line 32, in __set__
    obj.__dict__[self.field.name] = self.field.to_python(value)
  File "P:\home\mark\Python\intomec\..\intomec\tenq\models.py", line 42, in to_python
    return pickle.loads(value)
  File "c:\Python25\lib\pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "c:\Python25\lib\pickle.py", line 858, in load
    dispatch[key](self)
KeyError: '\x00'
>>>

the class= part (right before the Traceback) is printed by the class right before the problem in pickle.loadds

Why is the db output converted into unicode before to_python is called? the KeyError is because the Unpickler expects a bytestream, not a unicode stream.....

[Jacob]

I suspect this is because you're putting pickle data into a TextField or CharField, which expects text, not binary. If you're storing pickled data, you'll want to write a custom field (​http://docs.djangoproject.com/en/dev/howto/custom-model-fields/).

If you're not using a TextField/CharField, please feel free to reopen.

[freeav8r]

The following works well for unpickling if you don't want to modify the field:

object = loads(pickledstring.encode('utf-8'))