Opened 15 years ago

Closed 14 years ago

Last modified 14 years ago

#11753 closed (fixed)

Q objects using callables won't combine in 2.4

Reported by: Alex Robbins Owned by:
Component: Database layer (models, ORM) Version: 1.1
Severity: Keywords: Q, mq
Cc: immel@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

This causes a stack trace in 2.4 only.

#in python2.4
from django.db.models import Q

def a():
    return None

Q(test=a)|Q(test=1)

This looks to be hitting the same python deepcopy bug as #5505.

This only happens if the callable parameter is first.
For example (using same function as above):

Q(test=1)|Q(test=1) #OK
Q(test=1)|Q(test=a) #OK
Q(test=a)|Q(test=1) #Bad
Q(test=a)|Q(test=a) #Bad

Attachments (2)

11753.diff​ (24.0 KB ) - added by Jeremy Dunck 14 years ago.
Attached patch copies in py2.6 copy.py and uses it only if the stdlib copy is effected by the bug.
ticket11753.diff​ (12.7 KB ) - added by Jeremy Dunck 14 years ago.
OK, here's an updated patch, still using copycompat, but just shimming in the FunctionType dispatch.

Download all attachments as: .zip

Change History (18)

comment:1 by Alex Robbins, 15 years ago

Yeah, the issue is here ​http://code.djangoproject.com/browser/django/trunk/django/db/models/query_utils.py#L154

Still trying to think of a solution.

A secondary bug is that you can't invert a Q object with a callable in 2.4 (it uses deepcopy too).

comment:2 by Alex Robbins, 15 years ago

Well, I found a hack around the issue.

If you use a function as the callable it blows up. However, if you use a class with a call method deepcopy behaves and everything works out ok.

from django.db.models import Q

class a(object):
    def __call__(self):
        return None

Q(test=a())|Q(test=1) # OK

Not sure why that fixes it (read:too lazy to wander through deepcopy), but if anyone else runs into this quirk hopefully this helps.

comment:3 by Alex Robbins, 15 years ago

Stacktrace for google.

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 174, in deepcopy
    y = copier(x, memo)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 241, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 174, in deepcopy
    y = copier(x, memo)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 248, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 204, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 336, in _reconstruct
    y = callable(*args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy_reg.py", line 92, in __newobj__
    return cls.__new__(cls, *args)
TypeError: function() takes at least 2 arguments (0 given)

comment:4 by Ramiro Morales, 14 years ago

Owner: changed from nobody to Ramiro Morales
Status: new → assigned

This seem to be causing test suite failures too as of r11745, runtests.py queries output follows:

======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ramiro/django/trunk/django/test/_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.queries.models.__test__.API_TESTS
  File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line unknown line number, in API_TESTS

----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Item.objects.filter(Q(tags=t1) & Q(tags=t2))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[51]>", line 1, in ?
        Item.objects.filter(Q(tags=t1) & Q(tags=t2))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 162, in __and__
        return self._combine(other, self.AND)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    query = Item.objects.exclude(creator__in=[a1, a2]).query
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[109]>", line 1, in ?
        query = Item.objects.exclude(creator__in=[a1, a2]).query
      File "/home/ramiro/django/trunk/django/db/models/manager.py", line 140, in exclude
        return self.get_query_set().exclude(*args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 503, in exclude
        return self._filter_or_exclude(True, *args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 512, in _filter_or_exclude
        clone.query.add_q(~Q(*args, **kwargs))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 165, in __invert__
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    qs = Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[111]>", line 1, in ?
        qs = Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    len([x[2] for x in qs.query.alias_map.values() if x[2] == query.LOUTER and qs.query.alias_refcount[x[1]]])
Expected:
    1
Got:
    0
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[167]>", line 1, in ?
        Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(Q(extra__note=n1)|Q(item__note=n3)).filter(id=a1.id)
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[168]>", line 1, in ?
        Author.objects.filter(Q(extra__note=n1)|Q(item__note=n3)).filter(id=a1.id)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Tag.objects.exclude(parent=t1, name='t3').order_by('name')
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[192]>", line 1, in ?
        Tag.objects.exclude(parent=t1, name='t3').order_by('name')
      File "/home/ramiro/django/trunk/django/db/models/manager.py", line 140, in exclude
        return self.get_query_set().exclude(*args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 503, in exclude
        return self._filter_or_exclude(True, *args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 512, in _filter_or_exclude
        clone.query.add_q(~Q(*args, **kwargs))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 165, in __invert__
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    n1.annotation_set.filter(Q(tag=t5) | Q(tag__children=t5) | Q(tag__children__children=t5))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[227]>", line 1, in ?
        n1.annotation_set.filter(Q(tag=t5) | Q(tag__children=t5) | Q(tag__children__children=t5))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(Q(item__note__extrainfo=e2)|Q(report=r1, name='xyz'))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[262]>", line 1, in ?
        Author.objects.filter(Q(item__note__extrainfo=e2)|Q(report=r1, name='xyz'))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(Q(report=r1, name='xyz')|Q(item__note__extrainfo=e2))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[263]>", line 1, in ?
        Author.objects.filter(Q(report=r1, name='xyz')|Q(item__note__extrainfo=e2))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Annotation.objects.filter(Q(tag__parent=t1)|Q(notes__note='n1', name='a1'))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[264]>", line 1, in ?
        Annotation.objects.filter(Q(tag__parent=t1)|Q(notes__note='n1', name='a1'))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[266]>", line 1, in ?
        Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    q = Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx)).query
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[268]>", line 1, in ?
        q = Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx)).query
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    len([x[2] for x in q.alias_map.values() if x[2] == q.LOUTER and q.alias_refcount[x[1]]])
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[269]>", line 1, in ?
        len([x[2] for x in q.alias_map.values() if x[2] == q.LOUTER and q.alias_refcount[x[1]]])
    NameError: name 'q' is not defined


----------------------------------------------------------------------
Ran 6 tests in 1.178s

FAILED (failures=1)

System is a Debian Etch one, versions are:

$ dpkg -l python libsqlite3-0 python-pysqlite2
||/ Name                          Version                       Description
+++-=============================-=============================-==========================================================================
ii  libsqlite3-0                  3.3.8-1.1                     SQLite 3 shared library
ii  python                        2.4.4-2                       An interactive high-level object-oriented language (default version)
ii  python-pysqlite2              2.3.2-2                       python interface to SQLite 3

comment:5 by Ramiro Morales, 14 years ago

Owner: Ramiro Morales removed
Status: assigned → new
Triage Stage: Unreviewed → Accepted

comment:6 by anonymous, 14 years ago

Cc: immel@… added

comment:7 by Ramiro Morales, 14 years ago

bisection method shows this regression under python 2.4 was introduced in r11732.

by Jeremy Dunck, 14 years ago

Attachment: 11753.diff​ added

Attached patch copies in py2.6 copy.py and uses it only if the stdlib copy is effected by the bug.

comment:8 by Jeremy Dunck, 14 years ago

Keywords: mq added

Attached patch from Alex Robbins, thanks.

comment:9 by Ramiro Morales, 14 years ago

Has patch: set
Owner: set to Ramiro Morales

comment:10 by Alex Robbins, 14 years ago

Ramiro,
(We worked on this feature at a sprint over the weekend.)

The deepcopy bug is triggered whenever deepcopy tries to copy an object that has a function as an attribute. (Deepcopy works with methods, but not with unbound function attributes.) All of the test case failures come from using a Model instance inside a Q object. Perhaps r11732 somehow added an unbound function to model instances? (I looked at that revision, but it isn't immediately obvious to me if it might have done that.)

Adding the whole copy module from 2.6 seemed extreme to me at first. I tried modifying Q objects so that the init method changed and function arguments into classes with call methods instead. I had that working at one point, but it didn't fix the test cases. The model instances had callables, but not at the top level. Deepcopy was recursive, so the fix would have to recurse over the whole object also. At that point, we had to write a function that dynamically made classes (something that didn't quite work when we abandoned it) and also recursed over objects, avoiding circular refs and such.

jdunck looked into messing with copy_reg to register a new handler for functions. However, functions are one of the standard types, and copy_reg doesn't allow you to register new handlers for them.

In the end, we gave up and just added copy from 2.6. It seemed the simplest way to fix the problem.

Alex

comment:11 by Ramiro Morales, 14 years ago

Owner: Ramiro Morales removed

comment:12 by Jacob, 14 years ago

I'm really reluctant to include yet another backported bit of Python in django.utils -- I think that should be a last resort. Has anyone tracked down the Python bug in question? Or the patch that fixed it? Perhaps we can monkeypatch copy accordingly?

comment:13 by Jeremy Dunck, 14 years ago

jezdez asked the same question. :-)

r42573 | guido.van.rossum | 2006-02-25 16:38:04 -0600 (Sat, 25 Feb 2006) | 8 lines

- Patch 1433928:
  - The copy module now "copies" function objects (as atomic objects).
  - dict.__getitem__ now looks for a __missing__ hook before raising
    KeyError.
  - Added a new type, defaultdict, to the collections module.
    This uses the new __missing__ hook behavior added to dict (see above).

Basically, it fixes the problem by adding the type to an internal data structure. 

 33121     loewis d[types.BuiltinFunctionType] = _deepcopy_atomic
 42573 guido.van.rossum d[types.FunctionType] = _deepcopy_atomic

"d" there is an alias for copy._deepcopy_dispatch.

I considered poking a value in there, but thought it was a bit presumptuous. Since you're asking about it, lemme see if just poking it in there actually fixes the problem.

comment:14 by Jeremy Dunck, 14 years ago

OK, I whacked this into django/init.py for a test, and it fixed py2.4:

import copy
import types
print "hacking copy"
copy._deepcopy_dispatch[types.FunctionType] = copy._deepcopy_atomic

Obviously, I could check to make sure that those keys didn't already exist before setting them.

Even so, where in Django's tree would be an appropriate place to make sure this gets run?

Also, Alex pointed out to me that there have been a few commits along the way to work around this bug in py2.4, and we should chase down those to remove the newly-unnecessary complexity.

by Jeremy Dunck, 14 years ago

Attachment: ticket11753.diff​ added

OK, here's an updated patch, still using copycompat, but just shimming in the FunctionType dispatch.

comment:15 by Jacob, 14 years ago

Resolution: → fixed
Status: new → closed

(In [11901]) Fixed #11753 - Q objects with callables no longer explode on Python 2.4. Thanks, Jeremy Dunck.

comment:16 by Jacob, 14 years ago

(In [11903]) [1.1.X] Fixed #11753 - Q objects with callables no longer explode on Python 2.4. Thanks, Jeremy Dunck.

Backport of [11901].

Note: See TracTickets for help on using tickets.
Back to Top