Unable to pickle Queryset

[wfagan@…]

I am getting an error while trying this code. I am using memcached as my backend. It looks like it isn't able to correctly pickle the queryset. This article makes me think it should be able to pickle them... ​http://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets

>>> places = Place.objects.all()
>>> cache.set("places", pickle.dumps(places))
>>> c = cache.get("places")
>>> pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

[wfagan@…]

The formatting got messed up on my last post. Hopefully this will look better.

>>> places = Place.objects.all()
>>> cache.set("places", pickle.dumps(places))
>>> c = cache.get("places")
>>> pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

[Alex Gaynor]

Please use preview. Also can you provide your models so we can reproduce.

[wfagan@…]

I have attached a test project that has the problem. Here is the code that causes the problem.

from pickle_test.main.models import Place
from django.contrib.auth.models import User
from django.core.cache import cache
import pickler

user = User.objects.create(username="test_user")
place = Place.objects.create(name="Test Place", address="Test Address", city="Wichita", state="KS", creator=user)

places = Place.objects.all()
cache.set("places", pickle.dumps(places))
c = cache.get("places")
q = pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

[Alex Gaynor]

Using the models.py you provided these tests pass fine for me:

__test__ = {"doctest": """
>>> from pickle import dumps, loads
>>> from django_test.test_apps.pickle.models import Place
>>> from django.contrib.auth.models import User
>>> from django.core.cache import cache

>>> list(loads(dumps(Place.objects.all()))) == list(Place.objects.all())
True
>>> user = User.objects.create(username="test_user")
>>> place = Place.objects.create(name="Test Place", address="Test Address",
...     city="Wichita", state="KS", creator=user)
>>> list(loads(dumps(Place.objects.all()))) == list(Place.objects.all())
True
>>> places = Place.objects.all()
>>> cache.set("places", dumps(places))
>>> c = cache.get("places")
>>> list(loads(c)) == list(places)
True
"""}

Since pickling is tested quite heavily in the Django regression tests, and I'm unable to reproduce I'm going to close this as worksforme. I suggest you consult either #django or the django-users mailing list for support debugging your issue.

[ShawnMilo]

For the benefit of others who may find this ticket in a search: The error could be due to the fact that pickle.dumps() returned a str, but after restoring the serialized value from the cache it was a unicode object, which pickle.loads() did not expect.

In this case, replacing pickle.loads(c) with pickle.loads(str(c)) may solve the problem.

[Jacob]

Milestone 1.1 deleted