Opened 15 years ago

Closed 15 years ago

Last modified 12 years ago

#10865 closed (worksforme)

Unable to pickle Queryset

Reported by: wfagan@… Owned by: nobody
Component: Uncategorized Version: dev
Severity: Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Alex Gaynor)

I am getting an error while trying this code. I am using memcached as my backend. It looks like it isn't able to correctly pickle the queryset. This article makes me think it should be able to pickle them... http://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets

>>> places = Place.objects.all()
>>> cache.set("places", pickle.dumps(places))
>>> c = cache.get("places")
>>> pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

Attachments (1)

pickle_test.tar.gz (6.6 KB ) - added by wfagan@… 15 years ago.

Download all attachments as: .zip

Change History (7)

comment:1 by wfagan@…, 15 years ago

The formatting got messed up on my last post. Hopefully this will look better.

>>> places = Place.objects.all()
>>> cache.set("places", pickle.dumps(places))
>>> c = cache.get("places")
>>> pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

comment:2 by Alex Gaynor, 15 years ago

Description: modified (diff)

Please use preview. Also can you provide your models so we can reproduce.

by wfagan@…, 15 years ago

Attachment: pickle_test.tar.gz added

comment:3 by wfagan@…, 15 years ago

I have attached a test project that has the problem. Here is the code that causes the problem.

from pickle_test.main.models import Place
from django.contrib.auth.models import User
from django.core.cache import cache
import pickler

user = User.objects.create(username="test_user")
place = Place.objects.create(name="Test Place", address="Test Address", city="Wichita", state="KS", creator=user)

places = Place.objects.all()
cache.set("places", pickle.dumps(places))
c = cache.get("places")
q = pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

comment:4 by Alex Gaynor, 15 years ago

Resolution: worksforme
Status: newclosed

Using the models.py you provided these tests pass fine for me:

__test__ = {"doctest": """
>>> from pickle import dumps, loads
>>> from django_test.test_apps.pickle.models import Place
>>> from django.contrib.auth.models import User
>>> from django.core.cache import cache

>>> list(loads(dumps(Place.objects.all()))) == list(Place.objects.all())
True
>>> user = User.objects.create(username="test_user")
>>> place = Place.objects.create(name="Test Place", address="Test Address",
...     city="Wichita", state="KS", creator=user)
>>> list(loads(dumps(Place.objects.all()))) == list(Place.objects.all())
True
>>> places = Place.objects.all()
>>> cache.set("places", dumps(places))
>>> c = cache.get("places")
>>> list(loads(c)) == list(places)
True
"""}

Since pickling is tested quite heavily in the Django regression tests, and I'm unable to reproduce I'm going to close this as worksforme. I suggest you consult either #django or the django-users mailing list for support debugging your issue.

comment:5 by ShawnMilo, 14 years ago

For the benefit of others who may find this ticket in a search: The error could be due to the fact that pickle.dumps() returned a str, but after restoring the serialized value from the cache it was a unicode object, which pickle.loads() did not expect.

In this case, replacing pickle.loads(c) with pickle.loads(str(c)) may solve the problem.

comment:6 by Jacob, 12 years ago

milestone: 1.1

Milestone 1.1 deleted

Note: See TracTickets for help on using tickets.
Back to Top