Code

Opened 6 years ago

Closed 6 years ago

Last modified 3 years ago

#8484 closed (duplicate)

Pickling of SortedDict classes needs more robustness

Reported by: ckelly Owned by: nobody
Component: Core (Other) Version: master
Severity: Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

I have a queryset that contains an extra() call with a select parameter containing a "straight" SQL call - no in place variables, etc.

when I attempt to cache this result and then subsequently retrieve this object from said cache, I get an "'SortedDict' object has no attribute 'keyOrder'" attribute error

>>> qs = RevisionableModel.objects.extra(select={"rm_count": "SELECT COUNT(*) from extra_regress_revisionablemodel"})
>>> cache.set("extra_test", qs) 
>>> new_qs = cache.get("extra_test")

I'm using a file-based cache in this scenario, but was also able to recreate it with a db table cache.

Traceback is as follows:

exception raised:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest extra_test.models.__test__.API_TESTS[27]>", line 1, in <module>
        new_qs = cache.get("extra_test")
      File "/usr/local/lib/python2.5/site-packages/django/core/cache/backends/filebased.py", line 50, in get
        return pickle.load(f)
      File "/usr/local/lib/python2.5/site-packages/django/utils/datastructures.py", line 76, in __setitem__
        if key not in self.keyOrder:
    AttributeError: 'SortedDict' object has no attribute 'keyOrder'

It looks like it has to do with Pickling, but I attempted to do a quick pickle/ unpickle on the QS, to no avail:

new_qs = pickle.loads(pickle.dumps(qs))

This is a regression introduced in Changeset 8426

I have attached the models.py file, which is a modified version of the tests/regressiontests/extra_regress/models.py (rev 8463) with the additional test discussed above.

Attachments (1)

models.py (3.9 KB) - added by ckelly 6 years ago.
extra_regress test models.py with additional extra() caching test

Download all attachments as: .zip

Change History (5)

Changed 6 years ago by ckelly

extra_regress test models.py with additional extra() caching test

comment:1 Changed 6 years ago by mtredinnick

  • Component changed from Uncategorized to Core framework
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

This is one of those cool bugs that won't fail for everybody. This is because the order in which attributes our restored during unpickling is the order in which they were pickled, which is dependent upon the order the keys from the class's attribute dictionary are returned. Since hashing orders vary depending on platform/OS, Python type (e.g. jython, CPython), and Python version, it only fails sometimes.

So the trick is to either force keyOrder to be pickled earlier than the other data (by writing a custom __getstate__ method) or by being able to handle restores when keyOrder isn't already set. The former intuitively feels a bit nicer (more robust) to me.

comment:2 Changed 6 years ago by mtredinnick

  • Summary changed from Caching QuerySet with extra(select) causes AttributeError to Pickling of SortedDict classes needs more robustness

This is really just about pickling SortedDict structures, so changing the title to something more self-descriptive.

comment:3 Changed 6 years ago by mtredinnick

  • Resolution set to duplicate
  • Status changed from new to closed

With the root cause established, this is now a dupe of #7496 and I like the patch in that ticket.

comment:4 Changed 3 years ago by jacob

  • milestone 1.0 deleted

Milestone 1.0 deleted

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.