Opened 16 years ago

Closed 14 years ago

#7204 closed (fixed)

QuerySet cloning can sometimes fail

Reported by: Dorian Grey <imgrey@…> Owned by: Jacob
Component: Core (Other) Version: dev
Severity: Keywords: qsrf-cleanup revision 7520
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: yes Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Ramiro Morales)

[Original bug report removed; see the second comment for the original report. --JKM]

Under some circumstances -- often involving select_related() and count() -- QuerySet.clone() can cause exceptions related to deepcopy(). These exceptions look like

TypeError: instancemethod expected at least 2 arguments, got 0

or

TypeError: function expected at least 2 arguments, got 0

The first one involving instancemethod is probably related to a Python issue: see http://bugs.python.org/issue1515.

The second involving function only seems to occur on Python 2.4 (and not 2.5), but since 1515 isn't fixed yet, it may be a seperate problem.

In either case, though, the root problem is that for some reason QuerySet is trying to clone instancemethods and unbound functions, which isn't exactly supported -- dunno what a copy of a function would do anyway. The likely fix will involve finding out where and why functions/methods are being coppied, and stop doing that.

Change History (12)

comment:1 by oyvind, 16 years ago

Resolution: invalid
Status: newclosed

Post a simple example showing count not working.

comment:2 by Dorian Grey <imgrey@…>, 16 years ago

Resolution: invalid
Status: closedreopened

complete moldels.py:

from django.db import models
from django.contrib.auth.models import User
from django.dispatch import dispatcher
from django.db.models import signals

def _add_attrs_and_methods(sender, instance, signal, *args, **kwargs):
    import types

    def can_change(self, post):
	if self == post.author:
	    return True
	return False

    instance.can_change = types.MethodType(can_change, instance, instance.__class__)

class Halb(models.Model):
    fk = models.ForeignKey(User)
    field = models.CharField(max_length=10)

dispatcher.connect(_add_attrs_and_methods, sender=User, signal=signals.post_init)
In [1]: from users.models import *

In [2]: u = User.objects.all()[0]

In [3]: Halb.objects.filter(fk=u).count()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/tmp/tst/<ipython console> 

/usr/lib/python2.4/site-packages/django/db/models/query.py in count(self)
    182             return len(self._result_cache)
    183 
--> 184         return self.query.get_count()
    185 
    186     def get(self, *args, **kwargs):

/usr/lib/python2.4/site-packages/django/db/models/sql/query.py in get_count(self)
    209         """
    210         from subqueries import CountQuery
--> 211         obj = self.clone()
    212         obj.clear_ordering(True)
    213         obj.clear_limits()

/usr/lib/python2.4/site-packages/django/db/models/sql/query.py in clone(self, klass, **kwargs)
    167         obj.select = self.select[:]
    168         obj.tables = self.tables[:]
--> 169         obj.where = deepcopy(self.where)
    170         obj.where_class = self.where_class
    171         obj.group_by = self.group_by[:]

/usr/lib/python2.4/copy.py in deepcopy(x, memo, _nil)
    183             copier = _getspecial(cls, "__deepcopy__")
    184             if copier:
--> 185                 y = copier(x, memo)
    186             else:
    187                 reductor = dispatch_table.get(cls)

/usr/lib/python2.4/site-packages/django/utils/tree.py in __deepcopy__(self, memodict)
     43         obj = Node(connector=self.connector, negated=self.negated)
     44         obj.__class__ = self.__class__
---> 45         obj.children = deepcopy(self.children, memodict)
     46         obj.subtree_parents = deepcopy(self.subtree_parents, memodict)
     47         return obj

/usr/lib/python2.4/copy.py in deepcopy(x, memo, _nil)
    172     copier = _deepcopy_dispatch.get(cls)
    173     if copier:
--> 174         y = copier(x, memo)
    175     else:
    176         try:

/usr/lib/python2.4/copy.py in _deepcopy_list(x, memo)
    239     memo[id(x)] = y
    240     for a in x:
--> 241         y.append(deepcopy(a, memo))
    242     return y
    243 d[types.ListType] = _deepcopy_list

/usr/lib/python2.4/copy.py in deepcopy(x, memo, _nil)
    172     copier = _deepcopy_dispatch.get(cls)
    173     if copier:
--> 174         y = copier(x, memo)
    175     else:
    176         try:

/usr/lib/python2.4/copy.py in _deepcopy_tuple(x, memo)
    246     y = []
    247     for a in x:
--> 248         y.append(deepcopy(a, memo))
    249     d = id(x)
    250     try:

/usr/lib/python2.4/copy.py in deepcopy(x, memo, _nil)
    202                             raise Error(
    203                                 "un(deep)copyable object of type %s" % cls)
--> 204                 y = _reconstruct(x, rv, 1, memo)
    205 
    206     memo[d] = y

/usr/lib/python2.4/copy.py in _reconstruct(x, info, deep, memo)
    349     if state:
    350         if deep:
--> 351             state = deepcopy(state, memo)
    352         if hasattr(y, '__setstate__'):
    353             y.__setstate__(state)

/usr/lib/python2.4/copy.py in deepcopy(x, memo, _nil)
    172     copier = _deepcopy_dispatch.get(cls)
    173     if copier:
--> 174         y = copier(x, memo)
    175     else:
    176         try:

/usr/lib/python2.4/copy.py in _deepcopy_dict(x, memo)
    266     memo[id(x)] = y
    267     for key, value in x.iteritems():
--> 268         y[deepcopy(key, memo)] = deepcopy(value, memo)
    269     return y
    270 d[types.DictionaryType] = _deepcopy_dict

/usr/lib/python2.4/copy.py in deepcopy(x, memo, _nil)
    202                             raise Error(
    203                                 "un(deep)copyable object of type %s" % cls)
--> 204                 y = _reconstruct(x, rv, 1, memo)
    205 
    206     memo[d] = y

/usr/lib/python2.4/copy.py in _reconstruct(x, info, deep, memo)
    334     if deep:
    335         args = deepcopy(args, memo)
--> 336     y = callable(*args)
    337     memo[id(x)] = y
    338     if listiter is not None:

/usr/lib/python2.4/copy_reg.py in __newobj__(cls, *args)
     90 
     91 def __newobj__(cls, *args):
---> 92     return cls.__new__(cls, *args)
     93 
     94 def _slotnames(cls):

TypeError: instancemethod expected at least 2 arguments, got 0

comment:3 by Andy McCurdy <sedrik@…>, 16 years ago

Running r7494 of Django, I'm seeing the same behavior on python 2.4. python 2.5 seems to work as expected.

comment:4 by Jacob, 16 years ago

Description: modified (diff)
Summary: .count() not workingQuerySet cloning can sometimes fail
Triage Stage: UnreviewedAccepted

Changed title, description to more clearly explain what's going on here

comment:5 by George Vilches, 16 years ago

Keywords: qsrf-cleanup added

comment:6 by Ramiro Morales, 16 years ago

Description: modified (diff)

comment:7 by Jacob, 16 years ago

milestone: 1.0

comment:8 by Jacob, 16 years ago

Owner: changed from nobody to Jacob
Status: reopenednew

comment:9 by Jacob, 16 years ago

I think #7506 might have gotten to the bottom of this.

comment:10 by Malcolm Tredinnick, 16 years ago

Resolution: fixed
Status: newclosed

Fixed in [7773].

comment:11 by epiloque, 14 years ago

milestone: 1.0
Needs tests: set
Resolution: fixed
Status: closedreopened
Triage Stage: AcceptedUnreviewed

similar issue on Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) ( Ubuntu 9.10 package) with the same trace`
TypeError: function expected at least 2 arguments, got 0

no problem on 1.1

comment:12 by Karen Tracey, 14 years ago

Resolution: fixed
Status: reopenedclosed

Please open a new ticket for new problems. The fact that there's no problem on 1.1 indicates the original fix did work, and something subsequently has introduced the problem again. Tracking multiple occurrences of similar but in fact different due to different root causes problems in a single ticket gets confusing.

Note: See TracTickets for help on using tickets.
Back to Top