Opened 7 years ago

Closed 7 years ago

Last modified 4 years ago

#7813 closed (fixed)

Problem Pickling Querysets

Reported by: Ed Menendez <ed@…> Owned by: mtredinnick
Component: Database layer (models, ORM) Version: master
Severity: Keywords: queryset pickle select_related
Cc: 7944 Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: yes
Easy pickings: UI/UX:

Description

I saw a previously closed ticket 7506 for a related (no pun intended) problem. It seems like when we use select_related suddenly the queryset can't be pickled. This used to work back in May sometime. Currently using SVN 7944.

The models is attached.

# This works
ld = LeagueDivision.objects.all()
cache.set('test2', ld)

# This doesn't work
ld = LeagueDivision.objects.select_related('league_conference').all()
cache.set('test2', ld)

Traceback (most recent call last):

File "<console>", line 1, in <module>
File "C:\Python25\lib\site-packages\django\core\cache\backends

\memcached.py", line 35, in set

self._cache.set(smart_str(key), value, timeout or

self.default_timeout)

File "C:\Projects\pyapps\memcache.py", line 437, in set

return self._set("set", key, val, time, min_compress_len)

File "C:\Projects\pyapps\memcache.py", line 609, in _set

store_info = self._val_to_store_info(val, min_compress_len, key)

File "C:\Projects\pyapps\memcache.py", line 581, in

_val_to_store_info

pickler.dump(val)

File "C:\Python25\lib\copy_reg.py", line 71, in _reduce_ex

raise TypeError, "can't pickle %s objects" % base.name

TypeError: can't pickle function objects

The dev environment is Windows, Python 2.5.1 using runserver built in web server with memcache (also on Windows) and Postgres 8.2 on Windows.

  • Ed

Attachments (2)

models.py (923 bytes) - added by Ed Menendez <ed@…> 7 years ago.
Example model
clear_related_select_fields.diff (416 bytes) - added by jbronn 7 years ago.

Download all attachments as: .zip

Change History (6)

Changed 7 years ago by Ed Menendez <ed@…>

Example model

Changed 7 years ago by jbronn

comment:1 Changed 7 years ago by jbronn

  • Component changed from Uncategorized to Database wrapper
  • Has patch set
  • Keywords select_related added
  • Needs documentation unset
  • Needs tests set
  • Owner changed from nobody to mtredinnick
  • Patch needs improvement unset

I encountered this bug this past week and forgot to file a ticket. The problem is that when select_related is used, one of the internal Query data structures still contains references to Fields (some of which can't be pickled because of curried functions). This patch blanks out the related_select_fields structure prior to pickling, which should be OK because it's repopulated every time pre_sql_setup is called in as_sql. Malcolm will tell us if I'm wrong :)

comment:2 Changed 7 years ago by Simon Greenhill

  • milestone set to 1.0 beta
  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Accepted

comment:3 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [8053]) Fixed #7813 -- Allow pickling of Query classes that use select_related().
Based on a patch from Justin Bronn.

The test in this patch most likely breaks on Oracle. That's another issue.

comment:4 Changed 4 years ago by jacob

  • milestone 1.0 beta deleted

Milestone 1.0 beta deleted

Note: See TracTickets for help on using tickets.
Back to Top