Code

Opened 4 years ago

Closed 4 years ago

Last modified 22 months ago

#12121 closed (fixed)

deepcopy(model_instance) causes infinite recursion since r11681

Reported by: emulbreh Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords:
Cc: immel@… Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

deepcopy(mode_instance) causes infinite recursion in Model.__reduce__() since r11681.

$ python tests/runtests.py queries
      ...
    RuntimeError: maximum recursion depth exceeded
      ...

Attachments (2)

12121.reduce_deferred.diff (3.4 KB) - added by emulbreh 4 years ago.
t12121-r11683.diff (1.9 KB) - added by russellm 4 years ago.
Possible patch for weird reduce behavior.

Download all attachments as: .zip

Change History (20)

comment:1 Changed 4 years ago by russellm

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to worksforme
  • Status changed from new to closed

I ran the full test suite before I committed, and I just re-ran the queries test specifically - I don't see this failure.

comment:2 Changed 4 years ago by russellm

To clarify - I don't see this behaviour under MacOS, using Python 2.5, under SQLite, Postgres and MySQL. I'm willing to accept that you're seeing this problem, but we're going to need a lot more detail that you have provided so far.

comment:3 Changed 4 years ago by emulbreh

$ sqlite3 --version
3.6.5
$ svn up
At revision 11690.
$ export PYTHONPATH=`pwd`
$ export DJANGO_SETTINGS_MODULE=settings
$ cat settings.py 
DEBUG = True
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'test.db'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)
$ python django/bin/django-admin.py syncdb --noinput
$ python django/bin/django-admin.py shell
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) 
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.all()[0]
>>> ct
<ContentType: content type>
>>> from django.db.models import Q
>>> Q(ct=ct)
<django.db.models.query_utils.Q object at 0xdd0a30>
>>> Q(ct=ct) | Q(foo=True)
Traceback (most recent call last):
    ...
RuntimeError: maximum recursion depth exceeded
>>>
$ python tests/runtests.py queries
======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
...
$

comment:4 Changed 4 years ago by Alex

The test passes for me under sqlite3, python2.5. My SQLite is version 3.6.16 however.

comment:5 Changed 4 years ago by kmtracey

Perhaps it might shed some light if you did not trim all of the traceback, that might give a clue what is causing the recursion. FWIW I cannot recreate on Ubuntu nor Windows, with a somewhat lower level sqlite3:

kmt@lbox:~/django/trunk$ svn up
At revision 11690.
kmt@lbox:~/django/trunk$ svn diff
kmt@lbox:~/django/trunk$ sqlite3 --version
3.5.9
kmt@lbox:~/django/trunk$ export PYTHONPATH=`pwd`
kmt@lbox:~/django/trunk$ export DJANGO_SETTINGS_MODULE=settings
kmt@lbox:~/django/trunk$ cat settings.py
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'test.db'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)

kmt@lbox:~/django/trunk$ python django/bin/django-admin.py syncdb --noinput
kmt@lbox:~/django/trunk$ python django/bin/django-admin.py shell
Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.all()[0]
>>> ct
<ContentType: content type>
>>> from django.db.models import Q
>>> Q(ct=ct)
<django.db.models.query_utils.Q object at 0x84f09cc>
>>> Q(ct=ct) | Q(foo=True)
<django.db.models.query_utils.Q object at 0x84f0c4c>
>>> quit()
kmt@lbox:~/django/trunk$ python tests/runtests.py queries
----------------------------------------------------------------------
Ran 6 tests in 1.185s

OK
kmt@lbox:~/django/trunk$

I don't have any Macs to try.

comment:6 Changed 4 years ago by emulbreh

  • Resolution worksforme deleted
  • Status changed from closed to reopened

Searching for a python bug, I stumbled over a blog post from Malcolm: http://www.pointy-stick.com/blog/2009/03/23/yak-shaving-advanced-players/. Let me quote:

"Next morning, somebody reports an infinite recursion problem. That was very odd, since the code I wrote looked pretty correct in that area. The key seemed to be that they were using Python 2.4. Even creating a diff between the pickle module for Python 2.4 and the one for 2.5 was unenlightening, so the problem was deeper than that.

I'm not 100% sure if this is a bug in Python versions prior to 2.5, or just an undocumented side-effect. The end result is that if you write a __reduce__() method, you can't do the natural thing and dispatch to the same method on the superclass if you don't need to do anything special. At least, not in Python 2.3 or 2.4. Django supports those versions, so I had to work around.

Not a big problem, once I diagnosed it (turns out the existing test suite caught the problem when I ran it under Python 2.4), and fixed easily enough."

The ticket was #10547, the commit r10099. Apparently, my Apple "python 2.5" has some 2.4 bugs in it.

comment:7 Changed 4 years ago by emulbreh

Here's the requested traceback:

>>> Q(ct=ct) | Q(foo=True)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/query_utils.py", line 159, in __or__
    return self._combine(other, self.OR)
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/query_utils.py", line 154, in _combine
    obj = deepcopy(self)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 173, in deepcopy
    y = copier(memo)
  File "/Users/emulbreh/Projekte/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = deepcopy(self.children, memodict)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 227, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 234, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 181, in deepcopy
    rv = reductor(2)
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/base.py", line 356, in __reduce__
    return super(Model, self).__reduce__()
...
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/base.py", line 356, in __reduce__
    return super(Model, self).__reduce__()
RuntimeError: maximum recursion depth exceeded

Changed 4 years ago by emulbreh

comment:8 Changed 4 years ago by emulbreh

  • Has patch set
  • Patch needs improvement set

Just replacing super(Model, self).__reduce__() broke the cache tests.

comment:9 Changed 4 years ago by emulbreh

Note: The patch is bad. Model.__reduce__() cannot be removed. It's required for dynamically created models.

Changed 4 years ago by russellm

Possible patch for weird reduce behavior.

comment:10 follow-ups: Changed 4 years ago by russellm

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

comment:11 in reply to: ↑ 10 Changed 4 years ago by kmtracey

Replying to russellm:
I've got Python2.4 on both Ubuntu and Windows. Was able to see the infinite recursion failure once the early level of Python was identified as the cause, just confirmed the queries test passes with the patch.

comment:12 Changed 4 years ago by russellm

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [11692]) [1.1.X] Fixed #12121 -- Modified reduce on a model to avoid an infinite recursion problem that occurs on Python 2.4. Thanks to emulbreh for the report.

Backport of r11691 from trunk.

comment:13 in reply to: ↑ 10 ; follow-up: Changed 4 years ago by emulbreh

Replying to russellm:

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

It works, thanks.

comment:14 in reply to: ↑ 13 Changed 4 years ago by anonymous

Replying to emulbreh:

Replying to russellm:

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

It works, thanks.

Which means: with my broken Python 2.5, too.

comment:15 Changed 4 years ago by rfugger

I can confirm this bug on Red Hat under python 2.4.3. Where I see it is attempting to pickle an object for caching (using locmem cache) -- infinite recursion on Model.reduce.

Using python 2.5 works fine.

comment:16 Changed 4 years ago by russellm

@rfugger - are you reporting that this is *still* a problem? This should have been fixed by r11691; if it still exists for Python 2.4.3, then we have more work to do; a test case that triggers the infinite recursion would be most helpful.

comment:17 Changed 4 years ago by anonymous

  • Cc immel@… added

comment:18 Changed 22 months ago by Claude Paroz <claude@…>

In [146aff3bac974e56ec8cb597c2720d1cd4f77b26]:

Fixed #18590 - Reverted Python 2.4 workaround for Model pickling

Revert of 08d521efa0. Refs #10547, #12121.
Thanks Michal Petrucha for the report.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.