Opened 12 years ago

Closed 12 years ago

#17776 closed Bug (fixed)

DoesNotExist is not picklable

Reported by: Łukasz Langa Owned by: nobody
Component: Database layer (models, ORM) Version: 1.3
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

>>> import django
>>> django.contrib.auth.models.User.DoesNotExist
<class 'django.contrib.auth.models.DoesNotExist'>
>>> import pickle
>>> pickle.dumps(django.contrib.auth.models.User.DoesNotExist())
Traceback (most recent call last):
...
PicklingError: Can't pickle <class 'django.contrib.auth.models.DoesNotExist'>: it's not found as django.contrib.auth.models.DoesNotExist

The example uses contrib.auth.models.User but this is true for all models AFAICT.

The reason why picklability of this class is important is for multiprocessing exception handling, for instance in Celery tasks. Now every other exception in Celery gets a nice traceback in the logs but for DoesNotExist exceptions it's a useless:

Traceback (most recent call last):
 File "/home/virtualenv/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 199, in worker
   put((READY, (job, i, result)))
 File "/usr/lib/python2.7/multiprocessing/queues.py", line 392, in put
   return send(obj)
MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('LeagueMember matching query does not exist.',)>'. Reason: 'Can't pickle <class 'psl.league.models.DoesNotExist'>: attribute lookup psl.league.models.DoesNotExist failed'.

Change History (8)

comment:1 by Łukasz Langa, 12 years ago

Resolution: wontfix
Status: newclosed

Unfortunately, after further investigation it looks like this is an inherent limitation of pickling. The Python documentation states that "picklable functions and classes must be defined in the top level of a module".

Source: http://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled

So that's a WONTFIX, worth keeping for reference though.

comment:2 by Jeremy Dunck, 12 years ago

Resolution: wontfix
Status: closedreopened

Reopening because this problem affects the combination of Django, Celery, and Raven/Sentry, a very common thing indeed.

I am not sure if there's a fix available, but I'll look into it. Meantime, there was no discussion on this ticket and it seemed to be closed by reporter.

comment:3 by Jeremy Dunck, 12 years ago

Example failure in sentry:

(This is an Endorsement.DoesNotExist exception being captured under raven.)

<MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('Endorsement matching query does not exist.',)>'. Reason: 'Can't pickle <class 'candidates.models.DoesNotExist'>: attribute lookup candidates.models.DoesNotExist failed'.>

comment:4 by anonymous, 12 years ago

As I wrote in my WONTFIX explanation, this has to be fixed at the pickling level, specifically by introducing a new pickle protocol that can pickle inner classes. Good luck! ;)

comment:5 by ask@…, 12 years ago

You can make this pickleable by defining reduce, e.g.

class DoesNotExist(..):

     def __reduce__(self):
         return _unpickle_DoesNotExist, (self.app_label, self.model, self.args)


 def _unpickle_DoesNotExist(app_label, model, args):
     return get_model(app_label, model).DoesNotExist(*args) 

comment:6 by anonymous, 12 years ago

Exactly, there is at least one precedent in Django: r9272.

comment:7 by Luke Plant, 12 years ago

Triage Stage: UnreviewedAccepted

Assuming the fix as described above is uncomplicated, this seems worth fixing. Test and patch needed.

comment:8 by Luke Plant <L.Plant.98@…>, 12 years ago

Resolution: fixed
Status: reopenedclosed

In [a54a8bab0c6a96c03452040e92b2a3141695a363]:

Fixed #17776 - DoesNotExist is not picklable

Thanks to ambv for the report

Note: See TracTickets for help on using tickets.
Back to Top