python_2_unicode_compatible causes infinite recursion when super().__str__() is called

[Josh Crompton]

I have a class A which implements __str__() and is decorated with @python_2_unicode_compatible.

I have sub-class of A, B, and B 's implementation of __str__() calls super().__str__().

When I call b.__str__(), I get the following error: RuntimeError: maximum recursion depth exceeded while calling a Python object.

Partial traceback:

  File "/home/josh/Desktop/temp/djtest/local/lib/python2.7/site-packages/django/utils/encoding.py", line 42, in <lambda>
    klass.__str__ = lambda self: self.__unicode__().encode('utf-8')
  File "recursion.py", line 27, in __str__
    super().__str__(),
  File "/home/josh/Desktop/temp/djtest/local/lib/python2.7/site-packages/django/utils/encoding.py", line 42, in <lambda>
    klass.__str__ = lambda self: self.__unicode__().encode('utf-8')
  File "recursion.py", line 27, in __str__
    super().__str__(),

The docs (​https://docs.djangoproject.com/en/dev/ref/utils/#django.utils.encoding.python_2_unicode_compatible) imply that the decorator should be applied to *any* class implementing __str__()

[Tim Graham]

The attached script prints "foo bar" for me on Python 3.4. Is it supposed to reproduce the infinite recursion?

[Josh Crompton]

Apologies, I should have specified that this problem only occurs with Python 2 (I have tried only with 2.7.6). My understanding is that python_2_unicode_compatible is a noop in Python 3.

I also notice now that python_2_unicode_compatible is actually in the six library, so it might be more appropriate for me to make a ticket for that project than here.

[Tim Graham]

Can you omit @python_2_unicode_compatible on the subclass?

[Josh Crompton]

Unfortunately not, if you include a unicode character in Bar.extra and remove the decorator, you'll get something like:

Traceback (most recent call last):
  File "./recursion.py", line 34, in <module>
    print(b)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1546' in position 4: ordinal not in range(128)

[Josh Crompton]

I believe there's a related infinite-recursion problem on saving models in certain circumstances but I haven't had time to extract a useful example from the project in which I'm seeing it. At this point, I think it's a problem with the implementation of __str__ in django.db.models.base.Model.

[Aymeric Augustin]

I don't understand how super().__str__() can work as super() without arguments is illegal in Python 2.

[Tim Graham]

I think the future library does some monkeypatching to allow it to work, but it would be interesting to know if this issue can be reproduced without it.

[Simon Charette]

I reproduce with the correct super() call on Python 2.7:

In [1]: from django.utils.six import python_2_unicode_compatible

In [2]: @python_2_unicode_compatible
class A(object):
    def __str__(self):
        return str('a')
   ...:     

In [3]: str(A())
Out[3]: 'a'

In [4]: unicode(A())
Out[4]: u'a'

In [5]: class B(A):
    def __str__(self):
        return str('b') + super(B, self).__str__()
   ...:     

In [6]: str(B())
Out[6]: 'ba'

In [7]: unicode(B())  # Notice how B.__str__() isn't called.
Out[7]: u'a'

In [8]: @python_2_unicode_compatible
class C(A):
    def __str__(self):
        return str('c') + super(C, self).__str__()
   ...:     

In [9]: str(C())
--------------------------------------------------------------------------
...
<ipython-input-8-efd21456ef94> in __str__(self)
      2 class C(A):
      3     def __str__(self):
----> 4         return str('c') + super(C, self).__str__()
      5 

django/django/utils/six.pyc in <lambda>(self)
    810                              klass.__name__)
    811         klass.__unicode__ = klass.__str__
--> 812         klass.__str__ = lambda self: self.__unicode__().encode('utf-8')
    813     return klass
    814 

RuntimeError: maximum recursion depth exceeded in __subclasscheck__
In [10]: unicode(C())
--------------------------------------------------------------------------
...
<ipython-input-8-efd21456ef94> in __str__(self)
      2 class C(A):
      3     def __str__(self):
----> 4         return str('c') + super(C, self).__str__()
      5 

django/django/utils/six.pyc in <lambda>(self)
    810                              klass.__name__)
    811         klass.__unicode__ = klass.__str__
--> 812         klass.__str__ = lambda self: self.__unicode__().encode('utf-8')
    813     return klass
    814 

RuntimeError: maximum recursion depth exceeded in __subclasscheck__

[Josh Crompton]

Replying to timgraham:

I think the future library does some monkeypatching to allow it to work, but it would be interesting to know if this issue can be reproduced without it.

Correct, I should have mentioned that in the ticket, sorry. As @charettes demos above, it's reproducible with the normal Python2-style super call.

[Aymeric Augustin]

@python_2_unicode_compatible changes the implementation of __str__ and __unicode__ on Python 2. If you want to call the __str__ method you defined in the parent class, you have to call super(B, self).__unicode__().

The following works both on Python 2 and 3:

from __future__ import print_function, unicode_literals

from django.utils.six import PY3, python_2_unicode_compatible

@python_2_unicode_compatible
class A(object):
    def __str__(self):
        return 'a'

@python_2_unicode_compatible
class B(A):
    def __str__(self):
        text_method = '__str__' if PY3 else '__unicode__'
        return getattr(super(B, self), text_method)() + 'b'

print(str(B()))

This version is more readable:

from __future__ import print_function, unicode_literals

from django.utils.six import PY3, python_2_unicode_compatible

@python_2_unicode_compatible
class A(object):
    def text(self):
        return 'a'
    __str__ = text

@python_2_unicode_compatible
class B(A):
    def text(self):
        return super(B, self).text() + 'b'
    __str__ = text

print(str(B()))

This isn't obvious but I can't see a way around it. (I'm the original author of this decorator which was added to six later.)