Pickling of SortedDict classes needs more robustness

[ckelly]

I have a queryset that contains an extra() call with a select parameter containing a "straight" SQL call - no in place variables, etc.

when I attempt to cache this result and then subsequently retrieve this object from said cache, I get an "'SortedDict' object has no attribute 'keyOrder'" attribute error

>>> qs = RevisionableModel.objects.extra(select={"rm_count": "SELECT COUNT(*) from extra_regress_revisionablemodel"})
>>> cache.set("extra_test", qs) 
>>> new_qs = cache.get("extra_test")

I'm using a file-based cache in this scenario, but was also able to recreate it with a db table cache.

Traceback is as follows:

exception raised:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest extra_test.models.__test__.API_TESTS[27]>", line 1, in <module>
        new_qs = cache.get("extra_test")
      File "/usr/local/lib/python2.5/site-packages/django/core/cache/backends/filebased.py", line 50, in get
        return pickle.load(f)
      File "/usr/local/lib/python2.5/site-packages/django/utils/datastructures.py", line 76, in __setitem__
        if key not in self.keyOrder:
    AttributeError: 'SortedDict' object has no attribute 'keyOrder'

It looks like it has to do with Pickling, but I attempted to do a quick pickle/ unpickle on the QS, to no avail:

new_qs = pickle.loads(pickle.dumps(qs))

This is a regression introduced in Changeset 8426

I have attached the models.py file, which is a modified version of the tests/regressiontests/extra_regress/models.py (rev 8463) with the additional test discussed above.

[ckelly]

extra_regress test models.py with additional extra() caching test

[Malcolm Tredinnick]

This is one of those cool bugs that won't fail for everybody. This is because the order in which attributes our restored during unpickling is the order in which they were pickled, which is dependent upon the order the keys from the class's attribute dictionary are returned. Since hashing orders vary depending on platform/OS, Python type (e.g. jython, CPython), and Python version, it only fails sometimes.

So the trick is to either force keyOrder to be pickled earlier than the other data (by writing a custom __getstate__ method) or by being able to handle restores when keyOrder isn't already set. The former intuitively feels a bit nicer (more robust) to me.

[Malcolm Tredinnick]

This is really just about pickling SortedDict structures, so changing the title to something more self-descriptive.

[Malcolm Tredinnick]

With the root cause established, this is now a dupe of #7496 and I like the patch in that ticket.

[Jacob]

Milestone 1.0 deleted