Opened 8 years ago

Closed 8 years ago

Last modified 5 years ago

#7813 closed (fixed)

Problem Pickling Querysets

Reported by: Ed Menendez <ed@…> Owned by: Malcolm Tredinnick
Component: Database layer (models, ORM) Version: master
Severity: Keywords: queryset pickle select_related
Cc: 7944 Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: yes
Easy pickings: UI/UX:


I saw a previously closed ticket 7506 for a related (no pun intended) problem. It seems like when we use select_related suddenly the queryset can't be pickled. This used to work back in May sometime. Currently using SVN 7944.

The models is attached.

# This works
ld = LeagueDivision.objects.all()
cache.set('test2', ld)

# This doesn't work
ld = LeagueDivision.objects.select_related('league_conference').all()
cache.set('test2', ld)

Traceback (most recent call last):

File "<console>", line 1, in <module>
File "C:\Python25\lib\site-packages\django\core\cache\backends

\", line 35, in set

self._cache.set(smart_str(key), value, timeout or


File "C:\Projects\pyapps\", line 437, in set

return self._set("set", key, val, time, min_compress_len)

File "C:\Projects\pyapps\", line 609, in _set

store_info = self._val_to_store_info(val, min_compress_len, key)

File "C:\Projects\pyapps\", line 581, in



File "C:\Python25\lib\", line 71, in _reduce_ex

raise TypeError, "can't pickle %s objects" %

TypeError: can't pickle function objects

The dev environment is Windows, Python 2.5.1 using runserver built in web server with memcache (also on Windows) and Postgres 8.2 on Windows.

  • Ed

Attachments (2) (923 bytes) - added by Ed Menendez <ed@…> 8 years ago.
Example model
clear_related_select_fields.diff (416 bytes) - added by jbronn 8 years ago.

Download all attachments as: .zip

Change History (6)

Changed 8 years ago by Ed Menendez <ed@…>

Attachment: added

Example model

Changed 8 years ago by jbronn

comment:1 Changed 8 years ago by jbronn

Component: UncategorizedDatabase wrapper
Has patch: set
Keywords: select_related added
Needs documentation: unset
Needs tests: set
Owner: changed from nobody to Malcolm Tredinnick
Patch needs improvement: unset

I encountered this bug this past week and forgot to file a ticket. The problem is that when select_related is used, one of the internal Query data structures still contains references to Fields (some of which can't be pickled because of curried functions). This patch blanks out the related_select_fields structure prior to pickling, which should be OK because it's repopulated every time pre_sql_setup is called in as_sql. Malcolm will tell us if I'm wrong :)

comment:2 Changed 8 years ago by Simon Greenhill

milestone: 1.0 beta
Patch needs improvement: set
Triage Stage: UnreviewedAccepted

comment:3 Changed 8 years ago by Malcolm Tredinnick

Resolution: fixed
Status: newclosed

(In [8053]) Fixed #7813 -- Allow pickling of Query classes that use select_related().
Based on a patch from Justin Bronn.

The test in this patch most likely breaks on Oracle. That's another issue.

comment:4 Changed 5 years ago by Jacob

milestone: 1.0 beta

Milestone 1.0 beta deleted

Note: See TracTickets for help on using tickets.
Back to Top