Code

Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#11753 closed (fixed)

Q objects using callables won't combine in 2.4

Reported by: alexr Owned by:
Component: Database layer (models, ORM) Version: 1.1
Severity: Keywords: Q, mq
Cc: immel@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

This causes a stack trace in 2.4 only.

#in python2.4
from django.db.models import Q

def a():
    return None

Q(test=a)|Q(test=1)

This looks to be hitting the same python deepcopy bug as #5505.

This only happens if the callable parameter is first.
For example (using same function as above):

Q(test=1)|Q(test=1) #OK
Q(test=1)|Q(test=a) #OK
Q(test=a)|Q(test=1) #Bad
Q(test=a)|Q(test=a) #Bad

Attachments (2)

11753.diff (24.0 KB) - added by jdunck 5 years ago.
Attached patch copies in py2.6 copy.py and uses it only if the stdlib copy is effected by the bug.
ticket11753.diff (12.7 KB) - added by jdunck 5 years ago.
OK, here's an updated patch, still using copycompat, but just shimming in the FunctionType dispatch.

Download all attachments as: .zip

Change History (18)

comment:1 Changed 5 years ago by alexr

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Yeah, the issue is here http://code.djangoproject.com/browser/django/trunk/django/db/models/query_utils.py#L154

Still trying to think of a solution.

A secondary bug is that you can't invert a Q object with a callable in 2.4 (it uses deepcopy too).

comment:2 Changed 5 years ago by alexr

Well, I found a hack around the issue.

If you use a function as the callable it blows up. However, if you use a class with a call method deepcopy behaves and everything works out ok.

from django.db.models import Q

class a(object):
    def __call__(self):
        return None

Q(test=a())|Q(test=1) # OK

Not sure why that fixes it (read:too lazy to wander through deepcopy), but if anyone else runs into this quirk hopefully this helps.

comment:3 Changed 5 years ago by alexr

Stacktrace for google.

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 174, in deepcopy
    y = copier(x, memo)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 241, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 174, in deepcopy
    y = copier(x, memo)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 248, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 204, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy.py", line 336, in _reconstruct
    y = callable(*args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/copy_reg.py", line 92, in __newobj__
    return cls.__new__(cls, *args)
TypeError: function() takes at least 2 arguments (0 given)

comment:4 Changed 5 years ago by ramiro

  • Owner changed from nobody to ramiro
  • Status changed from new to assigned

This seem to be causing test suite failures too as of r11745, runtests.py queries output follows:

======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ramiro/django/trunk/django/test/_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.queries.models.__test__.API_TESTS
  File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line unknown line number, in API_TESTS

----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Item.objects.filter(Q(tags=t1) & Q(tags=t2))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[51]>", line 1, in ?
        Item.objects.filter(Q(tags=t1) & Q(tags=t2))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 162, in __and__
        return self._combine(other, self.AND)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    query = Item.objects.exclude(creator__in=[a1, a2]).query
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[109]>", line 1, in ?
        query = Item.objects.exclude(creator__in=[a1, a2]).query
      File "/home/ramiro/django/trunk/django/db/models/manager.py", line 140, in exclude
        return self.get_query_set().exclude(*args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 503, in exclude
        return self._filter_or_exclude(True, *args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 512, in _filter_or_exclude
        clone.query.add_q(~Q(*args, **kwargs))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 165, in __invert__
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    qs = Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[111]>", line 1, in ?
        qs = Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    len([x[2] for x in qs.query.alias_map.values() if x[2] == query.LOUTER and qs.query.alias_refcount[x[1]]])
Expected:
    1
Got:
    0
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[167]>", line 1, in ?
        Author.objects.filter(id=a1.id).filter(Q(extra__note=n1)|Q(item__note=n3))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(Q(extra__note=n1)|Q(item__note=n3)).filter(id=a1.id)
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[168]>", line 1, in ?
        Author.objects.filter(Q(extra__note=n1)|Q(item__note=n3)).filter(id=a1.id)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Tag.objects.exclude(parent=t1, name='t3').order_by('name')
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[192]>", line 1, in ?
        Tag.objects.exclude(parent=t1, name='t3').order_by('name')
      File "/home/ramiro/django/trunk/django/db/models/manager.py", line 140, in exclude
        return self.get_query_set().exclude(*args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 503, in exclude
        return self._filter_or_exclude(True, *args, **kwargs)
      File "/home/ramiro/django/trunk/django/db/models/query.py", line 512, in _filter_or_exclude
        clone.query.add_q(~Q(*args, **kwargs))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 165, in __invert__
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    n1.annotation_set.filter(Q(tag=t5) | Q(tag__children=t5) | Q(tag__children__children=t5))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[227]>", line 1, in ?
        n1.annotation_set.filter(Q(tag=t5) | Q(tag__children=t5) | Q(tag__children__children=t5))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(Q(item__note__extrainfo=e2)|Q(report=r1, name='xyz'))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[262]>", line 1, in ?
        Author.objects.filter(Q(item__note__extrainfo=e2)|Q(report=r1, name='xyz'))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Author.objects.filter(Q(report=r1, name='xyz')|Q(item__note__extrainfo=e2))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[263]>", line 1, in ?
        Author.objects.filter(Q(report=r1, name='xyz')|Q(item__note__extrainfo=e2))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Annotation.objects.filter(Q(tag__parent=t1)|Q(notes__note='n1', name='a1'))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[264]>", line 1, in ?
        Annotation.objects.filter(Q(tag__parent=t1)|Q(notes__note='n1', name='a1'))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx))
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[266]>", line 1, in ?
        Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx))
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    q = Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx)).query
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[268]>", line 1, in ?
        q = Note.objects.filter(Q(extrainfo__author=a1)|Q(extrainfo=xx)).query
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 159, in __or__
        return self._combine(other, self.OR)
      File "/home/ramiro/django/trunk/django/db/models/query_utils.py", line 154, in _combine
        obj = deepcopy(self)
      File "copy.py", line 185, in deepcopy
        y = copier(x, memo)
      File "/home/ramiro/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
        obj.children = deepcopy(self.children, memodict)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 241, in _deepcopy_list
        y.append(deepcopy(a, memo))
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 335, in _reconstruct
        args = deepcopy(args, memo)
      File "copy.py", line 174, in deepcopy
        y = copier(x, memo)
      File "copy.py", line 248, in _deepcopy_tuple
        y.append(deepcopy(a, memo))
      File "copy.py", line 204, in deepcopy
        y = _reconstruct(x, rv, 1, memo)
      File "copy.py", line 336, in _reconstruct
        y = callable(*args)
      File "copy_reg.py", line 92, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: function() takes at least 2 arguments (0 given)
----------------------------------------------------------------------
File "/home/ramiro/django/trunk/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    len([x[2] for x in q.alias_map.values() if x[2] == q.LOUTER and q.alias_refcount[x[1]]])
Exception raised:
    Traceback (most recent call last):
      File "/home/ramiro/django/trunk/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[269]>", line 1, in ?
        len([x[2] for x in q.alias_map.values() if x[2] == q.LOUTER and q.alias_refcount[x[1]]])
    NameError: name 'q' is not defined


----------------------------------------------------------------------
Ran 6 tests in 1.178s

FAILED (failures=1) 

System is a Debian Etch one, versions are:

$ dpkg -l python libsqlite3-0 python-pysqlite2
||/ Name                          Version                       Description
+++-=============================-=============================-==========================================================================
ii  libsqlite3-0                  3.3.8-1.1                     SQLite 3 shared library
ii  python                        2.4.4-2                       An interactive high-level object-oriented language (default version)
ii  python-pysqlite2              2.3.2-2                       python interface to SQLite 3

comment:5 Changed 5 years ago by ramiro

  • Owner ramiro deleted
  • Status changed from assigned to new
  • Triage Stage changed from Unreviewed to Accepted

comment:6 Changed 5 years ago by anonymous

  • Cc immel@… added

comment:7 Changed 5 years ago by ramiro

bisection method shows this regression under python 2.4 was introduced in r11732.

Changed 5 years ago by jdunck

Attached patch copies in py2.6 copy.py and uses it only if the stdlib copy is effected by the bug.

comment:8 Changed 5 years ago by jdunck

  • Keywords Q, mq added; Q removed

Attached patch from Alex Robbins, thanks.

comment:9 Changed 5 years ago by ramiro

  • Has patch set
  • Owner set to ramiro

comment:10 Changed 5 years ago by alexr

Ramiro,
(We worked on this feature at a sprint over the weekend.)

The deepcopy bug is triggered whenever deepcopy tries to copy an object that has a function as an attribute. (Deepcopy works with methods, but not with unbound function attributes.) All of the test case failures come from using a Model instance inside a Q object. Perhaps r11732 somehow added an unbound function to model instances? (I looked at that revision, but it isn't immediately obvious to me if it might have done that.)

Adding the whole copy module from 2.6 seemed extreme to me at first. I tried modifying Q objects so that the init method changed and function arguments into classes with call methods instead. I had that working at one point, but it didn't fix the test cases. The model instances had callables, but not at the top level. Deepcopy was recursive, so the fix would have to recurse over the whole object also. At that point, we had to write a function that dynamically made classes (something that didn't quite work when we abandoned it) and also recursed over objects, avoiding circular refs and such.

jdunck looked into messing with copy_reg to register a new handler for functions. However, functions are one of the standard types, and copy_reg doesn't allow you to register new handlers for them.

In the end, we gave up and just added copy from 2.6. It seemed the simplest way to fix the problem.

Alex

comment:11 Changed 5 years ago by ramiro

  • Owner ramiro deleted

comment:12 Changed 5 years ago by jacob

I'm really reluctant to include yet another backported bit of Python in django.utils -- I think that should be a last resort. Has anyone tracked down the Python bug in question? Or the patch that fixed it? Perhaps we can monkeypatch copy accordingly?

comment:13 Changed 5 years ago by jdunck

jezdez asked the same question. :-)

r42573 | guido.van.rossum | 2006-02-25 16:38:04 -0600 (Sat, 25 Feb 2006) | 8 lines

- Patch 1433928:
  - The copy module now "copies" function objects (as atomic objects).
  - dict.__getitem__ now looks for a __missing__ hook before raising
    KeyError.
  - Added a new type, defaultdict, to the collections module.
    This uses the new __missing__ hook behavior added to dict (see above).

Basically, it fixes the problem by adding the type to an internal data structure.

 33121     loewis d[types.BuiltinFunctionType] = _deepcopy_atomic
 42573 guido.van.rossum d[types.FunctionType] = _deepcopy_atomic

"d" there is an alias for copy._deepcopy_dispatch.

I considered poking a value in there, but thought it was a bit presumptuous. Since you're asking about it, lemme see if just poking it in there actually fixes the problem.

comment:14 Changed 5 years ago by jdunck

OK, I whacked this into django/init.py for a test, and it fixed py2.4:

import copy
import types
print "hacking copy"
copy._deepcopy_dispatch[types.FunctionType] = copy._deepcopy_atomic

Obviously, I could check to make sure that those keys didn't already exist before setting them.

Even so, where in Django's tree would be an appropriate place to make sure this gets run?

Also, Alex pointed out to me that there have been a few commits along the way to work around this bug in py2.4, and we should chase down those to remove the newly-unnecessary complexity.

Changed 5 years ago by jdunck

OK, here's an updated patch, still using copycompat, but just shimming in the FunctionType dispatch.

comment:15 Changed 5 years ago by jacob

  • Resolution set to fixed
  • Status changed from new to closed

(In [11901]) Fixed #11753 - Q objects with callables no longer explode on Python 2.4. Thanks, Jeremy Dunck.

comment:16 Changed 5 years ago by jacob

(In [11903]) [1.1.X] Fixed #11753 - Q objects with callables no longer explode on Python 2.4. Thanks, Jeremy Dunck.

Backport of [11901].

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.