Add a warning that objects may not be picklable across Django versions
|Reported by:||Catalin Iacob||Owned by:||nobody|
|Has patch:||yes||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
I just upgraded from 1.4 to 1.5 and this took a while to track.
If one has a cache that persists across a Django version change (filesystem cache, memcached that is not restarted), when executing any view decorated with @cache_page unpickling the HttpResponse from the cache leads to strange errors. For example, when upgrading from 1.4 to 1.5 I get:
[24/May/2013 15:53:42] "GET / HTTP/1.1" 200 5 Traceback (most recent call last): File "/usr/lib64/python2.7/wsgiref/handlers.py", line 86, in run self.finish_response() File "/usr/lib64/python2.7/wsgiref/handlers.py", line 129, in finish_response self.close() File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 36, in close SimpleHandler.close(self) File "/usr/lib64/python2.7/wsgiref/handlers.py", line 257, in close self.result.close() File "/home/catalin/hacking/envs/cached-httpresponse-across-versions/lib/python2.7/site-packages/django/http/response.py", line 231, in close for closable in self._closable_objects: AttributeError: 'HttpResponse' object has no attribute '_closable_objects' [24/May/2013 15:53:42] "GET / HTTP/1.1" 500 59 ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 49852) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib64/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/home/catalin/hacking/envs/cached-httpresponse-across-versions/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 150, in __init__ super(WSGIRequestHandler, self).__init__(*args, **kwargs) File "/usr/lib64/python2.7/SocketServer.py", line 638, in __init__ self.handle() File "/usr/lib64/python2.7/wsgiref/simple_server.py", line 124, in handle handler.run(self.server.get_app()) File "/usr/lib64/python2.7/wsgiref/handlers.py", line 89, in run self.handle_error() File "/usr/lib64/python2.7/wsgiref/handlers.py", line 304, in handle_error self.finish_response() File "/usr/lib64/python2.7/wsgiref/handlers.py", line 127, in finish_response self.write(data) File "/usr/lib64/python2.7/wsgiref/handlers.py", line 210, in write self.send_headers() File "/usr/lib64/python2.7/wsgiref/handlers.py", line 267, in send_headers if not self.origin_server or self.client_is_modern(): File "/usr/lib64/python2.7/wsgiref/handlers.py", line 280, in client_is_modern return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9' TypeError: 'NoneType' object has no attribute '__getitem__' ----------------------------------------
This was at first quite perplexing because I checked response.py and
self._closable_objects gets assigned in
HttpResponseBase.__init__ which gets called by
HttpResponse.__init__ so it seemed impossible for an HttpResponse instance not to have the attribute. Unpickling doesn't normally call
__init__ but this is not at all obvious or easy to track.
There are similar issues but with a bit more clear stacktrace if downgrading from 1.5 to 1.4:
Traceback: File "/home/catalin/hacking/envs/cached-httpresponse-across-versions/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response 89. response = middleware_method(request) File "/home/catalin/hacking/envs/cached-httpresponse-across-versions/lib/python2.7/site-packages/django/middleware/cache.py" in process_request 147. response = self.cache.get(cache_key, None) File "/home/catalin/hacking/envs/cached-httpresponse-across-versions/lib/python2.7/site-packages/django/core/cache/backends/filebased.py" in get 41. return pickle.load(f) Exception Type: ImportError at / Exception Value: No module named response
I'm surprised nobody seems to have run into this so far, googling for AttributeError on _closable_objects and searching the other bugs didn't show anything. I would imagine it's reasonably common to upgrade without restarting memcached or to have a filesystem based cache on a development machine where you don't bother to install memcached. And this happens on every view with @cache_page.
I don't know what a good solution would be. I can imagine Django doesn't want to guarantee pickle/unpickle round trips across versions as those can be hard to maintain but I think there should at least be a warning in the upgrade notes (of every version): when upgrading Django you need to delete your cache.
Change History (9)
comment:1 Changed 3 years ago by
|Patch needs improvement:||unset|
comment:2 Changed 3 years ago by
|Component:||Core (Cache system) → Documentation|
|Triage Stage:||Unreviewed → Accepted|
comment:3 Changed 3 years ago by
|Summary:||HttpResponse objects cached with cache_page don't unpickle across Django version leading to strange errors → Add a warning that objects may not be picklable across Django versions|