deepcopy(model_instance) causes infinite recursion since r11681

[Johannes Dollinger]

deepcopy(mode_instance) causes infinite recursion in Model.__reduce__() since r11681.

$ python tests/runtests.py queries
      ...
    RuntimeError: maximum recursion depth exceeded
      ...

[Russell Keith-Magee]

I ran the full test suite before I committed, and I just re-ran the queries test specifically - I don't see this failure.

[Russell Keith-Magee]

To clarify - I don't see this behaviour under MacOS, using Python 2.5, under SQLite, Postgres and MySQL. I'm willing to accept that you're seeing this problem, but we're going to need a lot more detail that you have provided so far.

[Johannes Dollinger]

$ sqlite3 --version
3.6.5
$ svn up
At revision 11690.
$ export PYTHONPATH=`pwd`
$ export DJANGO_SETTINGS_MODULE=settings
$ cat settings.py 
DEBUG = True
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'test.db'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)
$ python django/bin/django-admin.py syncdb --noinput
$ python django/bin/django-admin.py shell
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) 
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.all()[0]
>>> ct
<ContentType: content type>
>>> from django.db.models import Q
>>> Q(ct=ct)
<django.db.models.query_utils.Q object at 0xdd0a30>
>>> Q(ct=ct) | Q(foo=True)
Traceback (most recent call last):
    ...
RuntimeError: maximum recursion depth exceeded
>>>
$ python tests/runtests.py queries
======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
...
$

[Alex Gaynor]

The test passes for me under sqlite3, python2.5. My SQLite is version 3.6.16 however.

[Karen Tracey]

Perhaps it might shed some light if you did not trim all of the traceback, that might give a clue what is causing the recursion. FWIW I cannot recreate on Ubuntu nor Windows, with a somewhat lower level sqlite3:

kmt@lbox:~/django/trunk$ svn up
At revision 11690.
kmt@lbox:~/django/trunk$ svn diff
kmt@lbox:~/django/trunk$ sqlite3 --version
3.5.9
kmt@lbox:~/django/trunk$ export PYTHONPATH=`pwd`
kmt@lbox:~/django/trunk$ export DJANGO_SETTINGS_MODULE=settings
kmt@lbox:~/django/trunk$ cat settings.py
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'test.db'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)

kmt@lbox:~/django/trunk$ python django/bin/django-admin.py syncdb --noinput
kmt@lbox:~/django/trunk$ python django/bin/django-admin.py shell
Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.all()[0]
>>> ct
<ContentType: content type>
>>> from django.db.models import Q
>>> Q(ct=ct)
<django.db.models.query_utils.Q object at 0x84f09cc>
>>> Q(ct=ct) | Q(foo=True)
<django.db.models.query_utils.Q object at 0x84f0c4c>
>>> quit()
kmt@lbox:~/django/trunk$ python tests/runtests.py queries
----------------------------------------------------------------------
Ran 6 tests in 1.185s

OK
kmt@lbox:~/django/trunk$

I don't have any Macs to try.

[Johannes Dollinger]

Searching for a python bug, I stumbled over a blog post from Malcolm: ​http://www.pointy-stick.com/blog/2009/03/23/yak-shaving-advanced-players/. Let me quote:

"Next morning, somebody reports an infinite recursion problem. That was very odd, since the code I wrote looked pretty correct in that area. The key seemed to be that they were using Python 2.4. Even creating a diff between the pickle module for Python 2.4 and the one for 2.5 was unenlightening, so the problem was deeper than that.

I'm not 100% sure if this is a bug in Python versions prior to 2.5, or just an undocumented side-effect. The end result is that if you write a __reduce__() method, you can't do the natural thing and dispatch to the same method on the superclass if you don't need to do anything special. At least, not in Python 2.3 or 2.4. Django supports those versions, so I had to work around.

Not a big problem, once I diagnosed it (turns out the existing test suite caught the problem when I ran it under Python 2.4), and fixed easily enough."

The ticket was #10547, the commit r10099. Apparently, my Apple "python 2.5" has some 2.4 bugs in it.

[Johannes Dollinger]

Here's the requested traceback:

>>> Q(ct=ct) | Q(foo=True)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/query_utils.py", line 159, in __or__
    return self._combine(other, self.OR)
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/query_utils.py", line 154, in _combine
    obj = deepcopy(self)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 173, in deepcopy
    y = copier(memo)
  File "/Users/emulbreh/Projekte/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = deepcopy(self.children, memodict)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 227, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 234, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 181, in deepcopy
    rv = reductor(2)
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/base.py", line 356, in __reduce__
    return super(Model, self).__reduce__()
...
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/base.py", line 356, in __reduce__
    return super(Model, self).__reduce__()
RuntimeError: maximum recursion depth exceeded

[Johannes Dollinger]

Just replacing super(Model, self).__reduce__() broke the cache tests.

[Johannes Dollinger]

Note: The patch is bad. Model.__reduce__() cannot be removed. It's required for dynamically created models.

[Russell Keith-Magee]

Possible patch for weird reduce behavior.

[Russell Keith-Magee]

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

[Karen Tracey]

Replying to russellm:
I've got Python2.4 on both Ubuntu and Windows. Was able to see the infinite recursion failure once the early level of Python was identified as the cause, just confirmed the queries test passes with the patch.

[Russell Keith-Magee]

(In [11692]) [1.1.X] Fixed #12121 -- Modified reduce on a model to avoid an infinite recursion problem that occurs on Python 2.4. Thanks to emulbreh for the report.

Backport of r11691 from trunk.

[Johannes Dollinger]

Replying to russellm:

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

It works, thanks.

[anonymous]

Replying to emulbreh:

Replying to russellm:

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

It works, thanks.

Which means: with my broken Python 2.5, too.

[Ryan Fugger]

I can confirm this bug on Red Hat under python 2.4.3. Where I see it is attempting to pickle an object for caching (using locmem cache) -- infinite recursion on Model.reduce.

Using python 2.5 works fine.

[Russell Keith-Magee]

@rfugger - are you reporting that this is *still* a problem? This should have been fixed by r11691; if it still exists for Python 2.4.3, then we have more work to do; a test case that triggers the infinite recursion would be most helpful.

[Claude Paroz <claude@…>]

In [146aff3bac974e56ec8cb597c2720d1cd4f77b26]:

Fixed #18590 - Reverted Python 2.4 workaround for Model pickling

Revert of 08d521efa0. Refs #10547, #12121.
Thanks Michal Petrucha for the report.