Opened 7 years ago

Closed 7 years ago

Last modified 4 years ago

#7496 closed (fixed)

Getting cached instance of SortedDict using db cache backend throws AttributeError

Reported by: John Huddleston <huddlej@…> Owned by: huddlej
Component: Core (Cache system) Version: master
Severity: Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:


Using database caching, I set a non-empty SortedDict instance in the cache. When I try to retrieve it, I get an AttributeError with the message "'SortedDict' object has no attribute 'keyOrder'". Here is the code to recreate the problem:

# Using database cache backend
from django.core.cache import cache
from django.utils.datastructures import SortedDict
value = SortedDict()
value['1'] = 1
cache.set('my_dict', value)

I looked into the database cache backend and traced the problem to the pickling of the SortedDict instance which can be recreated like this:

import base64
import pickle
from django.utils.datastructures import SortedDict
value = SortedDict()
value['1'] = 1
encoded = base64.encodestring(pickle.dumps(value, 2)).strip()
decoded = pickle.loads(base64.decodestring(encoded))

The call to pickle.loads results in the AttributeError. If the pickling protocol is changed in dumps to 0, the call to loads works. The problem appears to be that protocol 2 calls SortedDict.__new__ instead of creating an empty SortedDict. When __new__ is called, __init__ is not being called so keyOrder is not being initialized.

Possible Solutions:

If this is really unexpected behavior, the problem could be solved by changing the pickle protocol in the database caching backend from 2 to 0. It looks like the database cache backend is the only backend using pickle protocol 2 while the local memory backend is using protocol 0.

Another solution would be to subclass SortedDict.__new__ so keyOrder is always set:

def __new__(cls, *args, **kwargs):
    instance = super(SortedDict, cls).__new__(cls, *args, **kwargs)
    instance.keyOrder = []
    return instance 

I have attached patches reflecting both of these solutions ( and, respectively).

Attachments (2) (690 bytes) - added by John Huddleston <huddlej@…> 7 years ago.
db cache backend patch (580 bytes) - added by John Huddleston <huddlej@…> 7 years ago.
SortedDict patch

Download all attachments as: .zip

Change History (6)

Changed 7 years ago by John Huddleston <huddlej@…>

db cache backend patch

Changed 7 years ago by John Huddleston <huddlej@…>

SortedDict patch

comment:1 Changed 7 years ago by huddlej

  • Needs documentation unset
  • Needs tests unset
  • Owner changed from nobody to huddlej
  • Patch needs improvement unset
  • Status changed from new to assigned

comment:2 Changed 7 years ago by ericholscher

  • milestone set to 1.0
  • Triage Stage changed from Unreviewed to Accepted

comment:3 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [8531]) Fixed #7496 -- It's now possible to pickle SortedDicts with pickle protocol 2
(used in caching). Thanks, John Huddleston.

comment:4 Changed 4 years ago by jacob

  • milestone 1.0 deleted

Milestone 1.0 deleted

Note: See TracTickets for help on using tickets.
Back to Top