Opened 12 years ago

Closed 8 years ago

#17917 closed Bug (fixed)

Pickling queryset with annotations on related fields with callable defaults fails

Reported by: jensadne@… Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: queryset, pickle, aggregation
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Given the following models:

class Person(models.Model):
    name = models.CharField(max_length=32)

class LogEntry(models.Model):
    person = models.ForeignKey(Person, related_name='log_entries')
    created = models.DateTimeField(default=datetime.now)

the following function fails with

"TypeError: expected string or Unicode object, NoneType found":
def pickle_persons_with_latest_logentry_time():
    qs = Person.objects.all().annotate(latest_logentry_time=Max('log_entries__created'))
    pickled = cPickle.dumps(qs)

Changing to using the pickle module instead of cPickle changes the exception to:

PicklingError: Can't pickle <built-in method now of type object at 0x7f3e064bb020>: it's not found as __main__.now

so this looks related to #13328

Tested with Django 1.3.1 on python 2.4.3 (Centos 5), Django 1.3.1 on python 2.7 (Fedora 14), and latest Django-SVN with python 2.7, and got the same result on all of them.

Change History (6)

comment:1 by Aymeric Augustin, 12 years ago

Triage Stage: UnreviewedAccepted

comment:2 by Aymeric Augustin, 12 years ago

Indeed, I could reproduce the problem. Pickling works without the annotation and crashes with the annotation.

>>> import pickle
>>> import cPickle
>>> from django.db.models import Max
>>> from test_app.models import Person

>>> qs = Person.objects.all()
>>> pickled = cPickle.dumps(qs)
>>> pickled = pickle.dumps(qs)

>>> qs = Person.objects.all().annotate(latest_logentry_time=Max('log_entries__created'))
>>> pickled = cPickle.dumps(qs)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: expected string or Unicode object, NoneType found
>>> pickled = pickle.dumps(qs)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 562, in save_tuple
    save(element)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <built-in method now of type object at 0x102ab8420>: it's not found as __main__.now

comment:3 by Anssi Kääriäinen, 11 years ago

Component: ORM aggregationDatabase layer (models, ORM)

comment:4 by Tim Graham, 8 years ago

Has patch: set

Fixed by f403653cf146384946e5c879ad2a351768ebc226 in Django 1.6.

PR to add a unit test. Annotating across a relation isn't required to trigger the exception.

comment:5 by Tim Graham <timograham@…>, 8 years ago

In ca43ee9:

Refs #17917 -- Added a test for pickling annotations on fields with callable defaults.

Fixed in f403653cf146384946e5c879ad2a351768ebc226.

comment:6 by Tim Graham, 8 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top