Opened 4 years ago

Closed 3 years ago

#17776 closed Bug (fixed)

DoesNotExist is not picklable

Reported by: ambv Owned by: nobody
Component: Database layer (models, ORM) Version: 1.3
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


>>> import django
>>> django.contrib.auth.models.User.DoesNotExist
<class 'django.contrib.auth.models.DoesNotExist'>
>>> import pickle
>>> pickle.dumps(django.contrib.auth.models.User.DoesNotExist())
Traceback (most recent call last):
PicklingError: Can't pickle <class 'django.contrib.auth.models.DoesNotExist'>: it's not found as django.contrib.auth.models.DoesNotExist

The example uses contrib.auth.models.User but this is true for all models AFAICT.

The reason why picklability of this class is important is for multiprocessing exception handling, for instance in Celery tasks. Now every other exception in Celery gets a nice traceback in the logs but for DoesNotExist exceptions it's a useless:

Traceback (most recent call last):
 File "/home/virtualenv/lib/python2.7/site-packages/celery/concurrency/processes/", line 199, in worker
   put((READY, (job, i, result)))
 File "/usr/lib/python2.7/multiprocessing/", line 392, in put
   return send(obj)
MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('LeagueMember matching query does not exist.',)>'. Reason: 'Can't pickle <class 'psl.league.models.DoesNotExist'>: attribute lookup psl.league.models.DoesNotExist failed'.

Change History (8)

comment:1 Changed 4 years ago by ambv

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to wontfix
  • Status changed from new to closed

Unfortunately, after further investigation it looks like this is an inherent limitation of pickling. The Python documentation states that "picklable functions and classes must be defined in the top level of a module".


So that's a WONTFIX, worth keeping for reference though.

comment:2 Changed 3 years ago by jdunck

  • Resolution wontfix deleted
  • Status changed from closed to reopened

Reopening because this problem affects the combination of Django, Celery, and Raven/Sentry, a very common thing indeed.

I am not sure if there's a fix available, but I'll look into it. Meantime, there was no discussion on this ticket and it seemed to be closed by reporter.

comment:3 Changed 3 years ago by jdunck

Example failure in sentry:

(This is an Endorsement.DoesNotExist exception being captured under raven.)

<MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('Endorsement matching query does not exist.',)>'. Reason: 'Can't pickle <class 'candidates.models.DoesNotExist'>: attribute lookup candidates.models.DoesNotExist failed'.>

comment:4 Changed 3 years ago by anonymous

As I wrote in my WONTFIX explanation, this has to be fixed at the pickling level, specifically by introducing a new pickle protocol that can pickle inner classes. Good luck! ;)

comment:5 Changed 3 years ago by ask@…

You can make this pickleable by defining reduce, e.g.

class DoesNotExist(..):

     def __reduce__(self):
         return _unpickle_DoesNotExist, (self.app_label, self.model, self.args)

 def _unpickle_DoesNotExist(app_label, model, args):
     return get_model(app_label, model).DoesNotExist(*args) 

comment:6 Changed 3 years ago by anonymous

Exactly, there is at least one precedent in Django: r9272.

comment:7 Changed 3 years ago by lukeplant

  • Triage Stage changed from Unreviewed to Accepted

Assuming the fix as described above is uncomplicated, this seems worth fixing. Test and patch needed.

comment:8 Changed 3 years ago by Luke Plant <L.Plant.98@…>

  • Resolution set to fixed
  • Status changed from reopened to closed

In [a54a8bab0c6a96c03452040e92b2a3141695a363]:

Fixed #17776 - DoesNotExist is not picklable

Thanks to ambv for the report

Note: See TracTickets for help on using tickets.
Back to Top