Code

Opened 5 years ago

Closed 5 years ago

Last modified 3 years ago

#10865 closed (worksforme)

Unable to pickle Queryset

Reported by: wfagan@… Owned by: nobody
Component: Uncategorized Version: master
Severity: Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description (last modified by Alex)

I am getting an error while trying this code. I am using memcached as my backend. It looks like it isn't able to correctly pickle the queryset. This article makes me think it should be able to pickle them... http://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets

>>> places = Place.objects.all()
>>> cache.set("places", pickle.dumps(places))
>>> c = cache.get("places")
>>> pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

Attachments (1)

pickle_test.tar.gz (6.6 KB) - added by wfagan@… 5 years ago.

Download all attachments as: .zip

Change History (7)

comment:1 Changed 5 years ago by wfagan@…

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

The formatting got messed up on my last post. Hopefully this will look better.

>>> places = Place.objects.all()
>>> cache.set("places", pickle.dumps(places))
>>> c = cache.get("places")
>>> pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

comment:2 Changed 5 years ago by Alex

  • Description modified (diff)

Please use preview. Also can you provide your models so we can reproduce.

Changed 5 years ago by wfagan@…

comment:3 Changed 5 years ago by wfagan@…

I have attached a test project that has the problem. Here is the code that causes the problem.

from pickle_test.main.models import Place
from django.contrib.auth.models import User
from django.core.cache import cache
import pickler

user = User.objects.create(username="test_user")
place = Place.objects.create(name="Test Place", address="Test Address", city="Wichita", state="KS", creator=user)

places = Place.objects.all()
cache.set("places", pickle.dumps(places))
c = cache.get("places")
q = pickle.loads(c)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
TypeError: __import__() argument 1 must be string without null bytes, not str

comment:4 Changed 5 years ago by Alex

  • Resolution set to worksforme
  • Status changed from new to closed

Using the models.py you provided these tests pass fine for me:

__test__ = {"doctest": """
>>> from pickle import dumps, loads
>>> from django_test.test_apps.pickle.models import Place
>>> from django.contrib.auth.models import User
>>> from django.core.cache import cache

>>> list(loads(dumps(Place.objects.all()))) == list(Place.objects.all())
True
>>> user = User.objects.create(username="test_user")
>>> place = Place.objects.create(name="Test Place", address="Test Address",
...     city="Wichita", state="KS", creator=user)
>>> list(loads(dumps(Place.objects.all()))) == list(Place.objects.all())
True
>>> places = Place.objects.all()
>>> cache.set("places", dumps(places))
>>> c = cache.get("places")
>>> list(loads(c)) == list(places)
True
"""}

Since pickling is tested quite heavily in the Django regression tests, and I'm unable to reproduce I'm going to close this as worksforme. I suggest you consult either #django or the django-users mailing list for support debugging your issue.

comment:5 Changed 4 years ago by ShawnMilo

For the benefit of others who may find this ticket in a search: The error could be due to the fact that pickle.dumps() returned a str, but after restoring the serialized value from the cache it was a unicode object, which pickle.loads() did not expect.

In this case, replacing pickle.loads(c) with pickle.loads(str(c)) may solve the problem.

comment:6 Changed 3 years ago by jacob

  • milestone 1.1 deleted

Milestone 1.1 deleted

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.