Opened 10 years ago

Closed 10 years ago

Last modified 7 years ago

#8484 closed (duplicate)

Pickling of SortedDict classes needs more robustness

Reported by: ckelly Owned by: nobody
Component: Core (Other) Version: master
Severity: Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


I have a queryset that contains an extra() call with a select parameter containing a "straight" SQL call - no in place variables, etc.

when I attempt to cache this result and then subsequently retrieve this object from said cache, I get an "'SortedDict' object has no attribute 'keyOrder'" attribute error

>>> qs = RevisionableModel.objects.extra(select={"rm_count": "SELECT COUNT(*) from extra_regress_revisionablemodel"})
>>> cache.set("extra_test", qs) 
>>> new_qs = cache.get("extra_test")

I'm using a file-based cache in this scenario, but was also able to recreate it with a db table cache.

Traceback is as follows:

exception raised:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.5/site-packages/django/test/", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest extra_test.models.__test__.API_TESTS[27]>", line 1, in <module>
        new_qs = cache.get("extra_test")
      File "/usr/local/lib/python2.5/site-packages/django/core/cache/backends/", line 50, in get
        return pickle.load(f)
      File "/usr/local/lib/python2.5/site-packages/django/utils/", line 76, in __setitem__
        if key not in self.keyOrder:
    AttributeError: 'SortedDict' object has no attribute 'keyOrder'

It looks like it has to do with Pickling, but I attempted to do a quick pickle/ unpickle on the QS, to no avail:

new_qs = pickle.loads(pickle.dumps(qs))

This is a regression introduced in Changeset 8426

I have attached the file, which is a modified version of the tests/regressiontests/extra_regress/ (rev 8463) with the additional test discussed above.

Attachments (1) (3.9 KB) - added by ckelly 10 years ago.
extra_regress test with additional extra() caching test

Download all attachments as: .zip

Change History (5)

Changed 10 years ago by ckelly

Attachment: added

extra_regress test with additional extra() caching test

comment:1 Changed 10 years ago by Malcolm Tredinnick

Component: UncategorizedCore framework
Triage Stage: UnreviewedAccepted

This is one of those cool bugs that won't fail for everybody. This is because the order in which attributes our restored during unpickling is the order in which they were pickled, which is dependent upon the order the keys from the class's attribute dictionary are returned. Since hashing orders vary depending on platform/OS, Python type (e.g. jython, CPython), and Python version, it only fails sometimes.

So the trick is to either force keyOrder to be pickled earlier than the other data (by writing a custom __getstate__ method) or by being able to handle restores when keyOrder isn't already set. The former intuitively feels a bit nicer (more robust) to me.

comment:2 Changed 10 years ago by Malcolm Tredinnick

Summary: Caching QuerySet with extra(select) causes AttributeErrorPickling of SortedDict classes needs more robustness

This is really just about pickling SortedDict structures, so changing the title to something more self-descriptive.

comment:3 Changed 10 years ago by Malcolm Tredinnick

Resolution: duplicate
Status: newclosed

With the root cause established, this is now a dupe of #7496 and I like the patch in that ticket.

comment:4 Changed 7 years ago by Jacob

milestone: 1.0

Milestone 1.0 deleted

Note: See TracTickets for help on using tickets.
Back to Top