Opened 7 years ago

Closed 7 years ago

Last modified 5 years ago

#12924 closed (fixed)

Pickling QuerySet with related fields fails

Reported by: Dmitry Shevchenko Owned by: Alex Gaynor
Component: Database layer (models, ORM) Version: 1.2-beta
Severity: Keywords:
Cc: dmishe@…, sj@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Assume we have two related models:

from django.db import models

class Group(models.Model):
    title = models.CharField(max_length=10)
        
class Event(models.Model):
    group = models.ForeignKey('Group')

Pickling queryset with filtering by group then fails:

>>>pickle.dumps(Event.objects.filter(group=1))
PicklingError: Can't pickle <function _curried at 0x101fdd5f0>: it's not found as django.utils.functional._curried

However, if i specify that i'm filtering on __pk lookup, then it suddenly works:

pickle.dumps(Event.objects.filter(group__pk=1))

Attachments (3)

translation-wtf-party.diff (5.3 KB) - added by Alex Gaynor 7 years ago.
Initial version, seems to be working, fixes all teh tickets about pickling translatable strings as well.
translation-wtf-party.2.diff (7.0 KB) - added by Alex Gaynor 7 years ago.
Added some forgotten files (tests), also updated some comments.
translation-wtf-party.3.diff (8.5 KB) - added by Alex Gaynor 7 years ago.
Killed some, now dead, code.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 7 years ago by Dmitry Shevchenko

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Version: 1.11.2-beta

comment:2 Changed 7 years ago by Dmitry Shevchenko

Ok found this evil function :)

In [61]: Event.objects.filter(group=1).query.where.children[0].children[0][0].field.related_query_name
Out[61]: <function _curried at 0x101fdd5f0>

comment:3 Changed 7 years ago by Russell Keith-Magee

milestone: 1.2
Triage Stage: UnreviewedAccepted

comment:4 Changed 7 years ago by Russell Keith-Magee

Component: UncategorizedDatabase layer (models, ORM)

comment:5 Changed 7 years ago by Alex Gaynor

Owner: changed from nobody to Alex Gaynor
Status: newassigned

Changed 7 years ago by Alex Gaynor

Attachment: translation-wtf-party.diff added

Initial version, seems to be working, fixes all teh tickets about pickling translatable strings as well.

comment:6 Changed 7 years ago by mrts

Let me point out that there's an important difference in behaviour in of delayed loading in Alex's patch. Namely, as with Alex's, Malcolm's code installs the reference to delayed_loader() as a proxy for every real translation function. But when delayed_loader() runs, it substitutes the references to itself with the actual functions from the imported real translation module in Malcolm's version. So it will be called exactly once and function calls will be fast subsequently.

As of now, Alex's version retains the references so that all calls to translation functions actually pass through delayed_loader(). As

  • calls to translation functions are frequent and
  • imports inside a function make calling it rather slow compared with a simple proxy function call (which is somewhat unexpected -- one would assume that only the first call to import is expensive and others cheap once the module is loaded -- but only timeit will tell as they say :) )

I recommend retaining Malcolm's optimized version. Currying is fine though and makes the code more readable IMHO.

See discussion at http://botland.oebfare.com/logger/django-dev/2010/3/20/3/ .

comment:7 Changed 7 years ago by Stephan Jaensch

Cc: sj@… added

Changed 7 years ago by Alex Gaynor

Added some forgotten files (tests), also updated some comments.

Changed 7 years ago by Alex Gaynor

Killed some, now dead, code.

comment:8 Changed 7 years ago by Russell Keith-Magee

Resolution: fixed
Status: assignedclosed

(In [12866]) Fixed #12769, #12924 -- Corrected the pickling of curried and lazy objects, which was preventing queries with translated or related fields from being pickled. And lo, Alex Gaynor didst slayeth the dragon.

comment:13 Changed 5 years ago by Jacob

milestone: 1.2

Milestone 1.2 deleted

Note: See TracTickets for help on using tickets.
Back to Top