Opened 13 years ago
Closed 13 years ago
#17776 closed Bug (fixed)
DoesNotExist is not picklable
Reported by: | Łukasz Langa | Owned by: | nobody |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | 1.3 |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Accepted | |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
>>> import django >>> django.contrib.auth.models.User.DoesNotExist <class 'django.contrib.auth.models.DoesNotExist'> >>> import pickle >>> pickle.dumps(django.contrib.auth.models.User.DoesNotExist()) Traceback (most recent call last): ... PicklingError: Can't pickle <class 'django.contrib.auth.models.DoesNotExist'>: it's not found as django.contrib.auth.models.DoesNotExist
The example uses contrib.auth.models.User
but this is true for all models AFAICT.
The reason why picklability of this class is important is for multiprocessing exception handling, for instance in Celery tasks. Now every other exception in Celery gets a nice traceback in the logs but for DoesNotExist exceptions it's a useless:
Traceback (most recent call last): File "/home/virtualenv/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 199, in worker put((READY, (job, i, result))) File "/usr/lib/python2.7/multiprocessing/queues.py", line 392, in put return send(obj) MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('LeagueMember matching query does not exist.',)>'. Reason: 'Can't pickle <class 'psl.league.models.DoesNotExist'>: attribute lookup psl.league.models.DoesNotExist failed'.
Change History (8)
comment:1 by , 13 years ago
Resolution: | → wontfix |
---|---|
Status: | new → closed |
comment:2 by , 13 years ago
Resolution: | wontfix |
---|---|
Status: | closed → reopened |
Reopening because this problem affects the combination of Django, Celery, and Raven/Sentry, a very common thing indeed.
I am not sure if there's a fix available, but I'll look into it. Meantime, there was no discussion on this ticket and it seemed to be closed by reporter.
comment:3 by , 13 years ago
Example failure in sentry:
(This is an Endorsement.DoesNotExist exception being captured under raven.)
<MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('Endorsement matching query does not exist.',)>'. Reason: 'Can't pickle <class 'candidates.models.DoesNotExist'>: attribute lookup candidates.models.DoesNotExist failed'.>
comment:4 by , 13 years ago
As I wrote in my WONTFIX explanation, this has to be fixed at the pickling level, specifically by introducing a new pickle protocol that can pickle inner classes. Good luck! ;)
comment:5 by , 13 years ago
You can make this pickleable by defining reduce, e.g.
class DoesNotExist(..): def __reduce__(self): return _unpickle_DoesNotExist, (self.app_label, self.model, self.args) def _unpickle_DoesNotExist(app_label, model, args): return get_model(app_label, model).DoesNotExist(*args)
comment:7 by , 13 years ago
Triage Stage: | Unreviewed → Accepted |
---|
Assuming the fix as described above is uncomplicated, this seems worth fixing. Test and patch needed.
comment:8 by , 13 years ago
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
Unfortunately, after further investigation it looks like this is an inherent limitation of pickling. The Python documentation states that "picklable functions and classes must be defined in the top level of a module".
Source: http://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled
So that's a WONTFIX, worth keeping for reference though.