Code

Opened 4 years ago

Closed 4 years ago

#13119 closed (worksforme)

Can't pickle queryset

Reported by: anentropic Owned by: nobody
Component: Database layer (models, ORM) Version: 1.2-beta
Severity: Keywords: querySet pickle
Cc: ego@… Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Was working ok until we upgraded from 1.1.1 to 1.2-beta-1 this morning.
Started getting the "can't pickle function object" error from this code:

class RatingDimensionManager(models.Manager):
    use_for_related_fields = True
    
    def get_for(self, obj):
        ct = ContentType.objects.get_for_model(obj)# cached by contentypes framework
        def from_db(obj):
            result = self.filter(content_type=ct.pk)
            cache.set('ratings:dimensions_for_type:%s:%s' % (hash(str(self.all().query)), ct.pk), result, settings.RATINGS_DIMENSIONS_FOR_CT_TTL)
            return result
        value = cache.get('ratings:dimensions_for_type:%s:%s' % (hash(str(self.all().query)), ct.pk), None)
        if value is None:
            value = from_db(obj)
        return value

class RatingDimension(models.Model):
    content_type = models.ForeignKey(ContentType)
    label = models.CharField(max_length=256)
    sort_weight = models.IntegerField(default=100)
    average = models.FloatField(editable=False, default=0)
    
    objects = RatingDimensionManager()
    
    def __unicode__(self):
        return self.label

    class Meta():
        ordering = ('sort_weight','label',)

Seems related to http://code.djangoproject.com/ticket/7506

Attachments (0)

Change History (8)

comment:1 Changed 4 years ago by russellm

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to worksforme
  • Status changed from new to closed

Works for me:

>>> import pickle
>>> pickle.dumps(RatingDimension.objects.all())
ccopy_reg
_reconstructor
p0
(cdjango.db.models.query
QuerySet
p1
...

Two suggestions for providing bug reports:

  1. Provide an example of the code that is failing. "Pickling" is a very vague operation, and as I've demonstrated, it does work under some circumstances. You need to give specific instructions for how to generate a failure. Remember that Django has a fairly extensive test suite, and there are lots of pickling tests. If there is a problem, it's more than likely going to be in edge case usage.
  2. An example is good, but a *minimal* example is better. Is the RatingDimensionManager necessary to stimulate this problem? What about the Meta? The unicode call? I don't doubt that these are necessary for your code, but they don't assist the debugging process.

Looking at your models, there's a good chance that this is a duplicate of #12924. I'm going to close this ticket worksforme (because it does). If it turns out that this isn't a duplicate of #12924, please reopen with a minimal example and a sample of how to induce the error.

comment:2 Changed 4 years ago by noisyboiler

Does not work for me with memcached

comment:3 Changed 4 years ago by anentropic

Thanks noisyboiler. It was the cache.set() call that was failing for me, and I am using memcached too.

However Russell is correct I believe, if I change this line:

result = self.filter(content_type=ct.pk)

to

result = self.filter(content_type__pk=ct.pk)

(as per #12924) then it starts working again. And also, that second form seems more correct anyway. Though as I said, it used to work in 1.1.1

comment:4 Changed 4 years ago by noisyboiler

Doesn't Pickle:
cars = Car.objects.filter(advertcarisnull=False)
Does Pickle:
cars = Car.objects.filter(advertcarpkisnull=False)

Both yield same results. Thank You Russell and anentropic.

comment:5 Changed 4 years ago by ashchristopher

  • Resolution worksforme deleted
  • Status changed from closed to reopened

This isn't really a solution to the problem, or if it is a solution to the problem then the documentation on QuerySets is wrong. Either way, I don't believe this is solved.

According to documentation, this it valid:

the_region = TripRegion.objects.all()[0]
guides = DestinationGuide.objects.active().filter(region=the_region)

And sure, it is a valid way to perform a QuerySet lookup, however in order to make use of say caching which needs to pickle the query set we shouldn't have to change the QS lookup to:

guides = estinationGuide.objects.active().filter(regionpk=the_region.pk)

comment:6 Changed 4 years ago by kmtracey

PLEASE use preview (and WikiFormatting). Most of the code posted in the previous few updates is completely indecipherable.

comment:7 Changed 4 years ago by ashchristopher

Fail.

*sigh*

This isn't really a solution to the problem, or if it is a solution to the problem then the documentation on Query Sets is incomplete. Either way, I don't believe this is solved.

    the_region = TripRegion.objects.all()[0]
    guides = DestinationGuide.objects.active().filter(region=the_region)   

This statement is valid, but will not work when pickled

    pickle.dumps(guides)

    TypeError::can't pickle function objects

In order to get it to work, you need to change it to:

    the_region = TripRegion.objects.all()[0]
    guides = DestinationGuide.objects.active().filter(region__pk=the_region.pk)   

I specifically get this problem when trying to cache query set results.

It seems weird that I would have to change the way I do a lookup depending on whether I needed to cache the results or not.

comment:8 Changed 4 years ago by russellm

  • Resolution set to worksforme
  • Status changed from reopened to closed

I can't reproduce the problem reported by @ashchristopher on r12974. I used the following test in the queryset_pickle regression test:

    def test_related_field_by_query(self):
        Group.objects.create(name="Ponies Who Own Maybachs")
        g = Group.objects.all()[0]
        self.assert_pickles(Event.objects.filter(group=g))

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.