Context Navigation

← Previous Ticket
Next Ticket →

#20993 closed Uncategorized (duplicate)

filtering with F(), then count()ing throws a pickle ValueError

Reported by:	Artscoop <artscoop93.info@…>	Owned by:	nobody
Component:	Database layer (models, ORM)	Version:	1.5
Severity:	Normal	Keywords:
Cc:		Triage Stage:	Unreviewed
Has patch:	no	Needs documentation:	no
Needs tests:	no	Patch needs improvement:	no
Easy pickings:	no	UI/UX:	no

Description

This

results = self.only('id').filter(user=user, topic__isnull=False, date__gte=F('topic__updated'))

works, but

total = self.only('id').filter(user=user, topic__isnull=False, date__gte=F('topic__updated')).count()

does not, and fails with a ValueError : ctypes objects containing pointers cannot be pickled. aggregate(), values_list() and annotate() also fail with the same error. date and topic__updated are DateTimeFields.

Change History (6)

comment:1 by Baptiste Mispelon, 11 years ago

Resolution:	→ worksforme
Status:	new → closed

Hi,

I tried reproducing your issue butI haven't had much success.

Here's the simplified model that I used:

from django.db import models

class Foo(models.Model):
    bar = models.DateTimeField()
    baz = models.DateTimeField()

With this model, both of these lines work fine:

Foo.objects.only('id').filter(bar__gte=F('baz'))
Foo.objects.only('id').filter(bar__gte=F('baz')).count()

Can you show us the models that you're using (or better yet, a simplified version that reproduces the problem)?
The full traceback would also be helpful in isolating the issue.

I'm closing this as worksforme for now. Please reopen this ticket with more information if you can.

Thanks.

comment:2 by Artscoop <artscoop93.info@…>, 11 years ago

Resolution:	worksforme
Status:	closed → new

Hi Baptiste,
Here they are

class TopicRead(models.Model):
    topic = models.ForeignKey('messaging.Topic', on_delete=models.CASCADE, verbose_name=_(u"Topic"))
    user = models.ForeignKey('auth.User', verbose_name=_(u"User"))
    date = models.DateTimeField(auto_now=True, db_index=True, verbose_name=_(u"Date"))
    objects = ReadManager()

    class Meta:
        unique_together = (('topic', 'user'),)
        verbose_name = _(u"read")
        verbose_name_plural = _(u"reads")
        app_label = 'messaging'

and Topic

class Topic(models.Model):
    author = models.ForeignKey('auth.User', null=True, verbose_name=_(u"Author"))
    topic = models.CharField(max_length=128, verbose_name=_(u"Topic"))
    started = models.DateTimeField(auto_now_add=True, verbose_name=_(u"Started"))
    updated = models.DateTimeField(auto_now=False, db_index=True, verbose_name=_(u"Updated"))
    deleted = models.BooleanField(default=False, db_index=True, verbose_name=_(u"Deleted"))
    closed = models.BooleanField(default=False, db_index=True, verbose_name=_(u"Fermé"))
    expires = models.DateTimeField(null=True, verbose_name=_(u"Expiry"))
    objects = TopicManager()

    def get_recipients(self, exclude=None):
        [...]

    def is_read(self, user):
        [...]

    def is_recipient(self, user):
        [...]

    def add(self, author, body, mail=True, request=None, important=False):
        [...]

    def close(self, administrative=False):
        [...]

    def __unicode__(self):
        return self.topic

    def get_absolute_url(self):
        [...]

    class Meta:
        verbose_name = _(u"topic")
        verbose_name_plural = _(u"topics")
        app_label = 'messaging'

comment:3 by Anssi Kääriäinen, 11 years ago

I have a feeling this could be about "deepcopy going too deep" issue, which isn't present in 1.6 any more. I don't have any concrete evidence of that, but deepcopy did go too deep when F() was involved in the query. See #16759 for the deepcopy removal.

comment:4 by Artscoop <artscoop93.info@…>, 11 years ago

Indeed, it has to do with deepcopy which is called 40 times for this query (not counting the private deepcopy and reconstruct), I forgot about the backtrace

  File "/home/django/continuous/lovelive/messaging/models/read.py", line 20, in get_read_count
    results = self.filter(user=user, topic__isnull=False, date__gte=F('date')).count()
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/db/models/query.py", line 388, in count
    return self.query.get_count(using=self.db)
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 408, in get_count
    obj = self.clone()
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 269, in clone
    obj.where = copy.deepcopy(self.where, memo=memo)
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 182, in deepcopy
    rv = reductor(2)

I am going to take a look at that ticket, thanks.

comment:5 by Artscoop <artscoop93.info@…>, 11 years ago

And indeed, you were right, after some deprecation fixes on a few places, I'm glad to tell you that it works without any flaw in Django 1.6b2.
Has this fix any chance to get backported one day ?

comment:6 by Marc Tamlyn, 11 years ago

Resolution:	→ duplicate
Status:	new → closed

It's very unlikely to be backported, according to our backwards compatibility policy - https://docs.djangoproject.com/en/dev/internals/release-process/#backwards-compatibility-policy

1.6 is on the horizon though, shouldn't be long to wait.

Note: See TracTickets for help on using tickets.

Download in other formats:

Issues

Context Navigation

#20993 closed Uncategorized (duplicate)

filtering with F(), then count()ing throws a pickle ValueError

Description

Change History (6)

comment:1 by Baptiste Mispelon, 11 years ago

comment:2 by Artscoop <artscoop93.info@…>, 11 years ago

comment:3 by Anssi Kääriäinen, 11 years ago

comment:4 by Artscoop <artscoop93.info@…>, 11 years ago

comment:5 by Artscoop <artscoop93.info@…>, 11 years ago

comment:6 by Marc Tamlyn, 11 years ago

Download in other formats:

Django Links

Learn More

Get Involved

Get Help

Follow Us

Support Us