Opened 4 years ago

Closed 4 years ago

#20993 closed Uncategorized (duplicate)

filtering with F(), then count()ing throws a pickle ValueError

Reported by: Artscoop <artscoop93.info@…> Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

This

results = self.only('id').filter(user=user, topic__isnull=False, date__gte=F('topic__updated'))

works, but

total = self.only('id').filter(user=user, topic__isnull=False, date__gte=F('topic__updated')).count()

does not, and fails with a ValueError : ctypes objects containing pointers cannot be pickled. aggregate(), values_list() and annotate() also fail with the same error. date and topic__updated are DateTimeFields.

Change History (6)

comment:1 Changed 4 years ago by Baptiste Mispelon

Resolution: worksforme
Status: newclosed

Hi,

I tried reproducing your issue butI haven't had much success.

Here's the simplified model that I used:

from django.db import models

class Foo(models.Model):
    bar = models.DateTimeField()
    baz = models.DateTimeField()

With this model, both of these lines work fine:

Foo.objects.only('id').filter(bar__gte=F('baz'))
Foo.objects.only('id').filter(bar__gte=F('baz')).count()

Can you show us the models that you're using (or better yet, a simplified version that reproduces the problem)?
The full traceback would also be helpful in isolating the issue.

I'm closing this as worksforme for now. Please reopen this ticket with more information if you can.

Thanks.

comment:2 Changed 4 years ago by Artscoop <artscoop93.info@…>

Resolution: worksforme
Status: closednew

Hi Baptiste,
Here they are

class TopicRead(models.Model):
    topic = models.ForeignKey('messaging.Topic', on_delete=models.CASCADE, verbose_name=_(u"Topic"))
    user = models.ForeignKey('auth.User', verbose_name=_(u"User"))
    date = models.DateTimeField(auto_now=True, db_index=True, verbose_name=_(u"Date"))
    objects = ReadManager()

    class Meta:
        unique_together = (('topic', 'user'),)
        verbose_name = _(u"read")
        verbose_name_plural = _(u"reads")
        app_label = 'messaging'

and Topic

class Topic(models.Model):
    author = models.ForeignKey('auth.User', null=True, verbose_name=_(u"Author"))
    topic = models.CharField(max_length=128, verbose_name=_(u"Topic"))
    started = models.DateTimeField(auto_now_add=True, verbose_name=_(u"Started"))
    updated = models.DateTimeField(auto_now=False, db_index=True, verbose_name=_(u"Updated"))
    deleted = models.BooleanField(default=False, db_index=True, verbose_name=_(u"Deleted"))
    closed = models.BooleanField(default=False, db_index=True, verbose_name=_(u"Fermé"))
    expires = models.DateTimeField(null=True, verbose_name=_(u"Expiry"))
    objects = TopicManager()

    def get_recipients(self, exclude=None):
        [...]

    def is_read(self, user):
        [...]

    def is_recipient(self, user):
        [...]

    def add(self, author, body, mail=True, request=None, important=False):
        [...]

    def close(self, administrative=False):
        [...]

    def __unicode__(self):
        return self.topic

    def get_absolute_url(self):
        [...]

    class Meta:
        verbose_name = _(u"topic")
        verbose_name_plural = _(u"topics")
        app_label = 'messaging'

comment:3 Changed 4 years ago by Anssi Kääriäinen

I have a feeling this could be about "deepcopy going too deep" issue, which isn't present in 1.6 any more. I don't have any concrete evidence of that, but deepcopy did go too deep when F() was involved in the query. See #16759 for the deepcopy removal.

comment:4 Changed 4 years ago by Artscoop <artscoop93.info@…>

Indeed, it has to do with deepcopy which is called 40 times for this query (not counting the private deepcopy and reconstruct), I forgot about the backtrace

  File "/home/django/continuous/lovelive/messaging/models/read.py", line 20, in get_read_count
    results = self.filter(user=user, topic__isnull=False, date__gte=F('date')).count()
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/db/models/query.py", line 388, in count
    return self.query.get_count(using=self.db)
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 408, in get_count
    obj = self.clone()
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 269, in clone
    obj.where = copy.deepcopy(self.where, memo=memo)
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/django/continuous/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 182, in deepcopy
    rv = reductor(2)

I am going to take a look at that ticket, thanks.

comment:5 Changed 4 years ago by Artscoop <artscoop93.info@…>

And indeed, you were right, after some deprecation fixes on a few places, I'm glad to tell you that it works without any flaw in Django 1.6b2.
Has this fix any chance to get backported one day ?

comment:6 Changed 4 years ago by Marc Tamlyn

Resolution: duplicate
Status: newclosed

It's very unlikely to be backported, according to our backwards compatibility policy - https://docs.djangoproject.com/en/dev/internals/release-process/#backwards-compatibility-policy

1.6 is on the horizon though, shouldn't be long to wait.

Note: See TracTickets for help on using tickets.
Back to Top