UnicodeDecode error when ChoiceField(w=RadioSelect) has Unicode choices

[Georgi Stanojevski <glisha gmail com>]

When I have a form like this, note the choices with unicode data I get an UnicodeDecodeError.

I attached a patch which just replaces the str method with unicode for RadioSelect widgets in django/newforms/forms.py +232.

It seems to work.

class aform(forms.Form):

    def __init__(self, *args, **kwargs):
        super(aform, self).__init__(*args, **kwargs)
        self.fields['aradio'].choices = [('1',u'јуникод'),('2','latin')]

    aradio = forms.ChoiceField(choices=(), widget=RadioSelect())

  f = aform()
  f.as_ul()

I get an UnicodeDecodeError.

exceptions.UnicodeDecodeError                        Traceback (most recent call last)

/home/webstrana/django/kajmakot/<ipython console>

/home/webstrana/django/django_src/django/newforms/forms.py in as_ul(self)
    146     def as_ul(self):
    147         "Returns this form rendered as HTML <li>s -- excluding the <ul></ul>."
--> 148         return self._html_output(u'<li>%(errors)s%(label)s %(field)s%(help_text)s</li>', u'<li>%s</li>', '</li>', u' %s', False)
    149
    150     def as_p(self):

/home/webstrana/django/django_src/django/newforms/forms.py in _html_output(self, normal_row, error_row, row_ender, help_text_html, errors_on_separate_row)
    127                 else:
    128                     help_text = u''
--> 129                 output.append(normal_row % {'errors': bf_errors, 'label': label, 'field': unicode(bf), 'help_text': help_text})
    130         if top_errors:
    131             output.insert(0, error_row % top_errors)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 80: ordinal not in range(128)

[Georgi Stanojevski <glisha gmail com>]

It replaces str with unicode when called on RadioSelect widgets

[Michael Radziej <mir@…>]

#3904 marked as duplicate

[Malcolm Tredinnick]

Since the object returned by RadioSelect.render() is a StrAndUnicode subclass, the change in the proposed patch shouldn't be necessary. Which suggests there is a deeper bug at work. I'm looking into it.

[Malcolm Tredinnick]

Wow... this did turn out to be the right patch (thanks, Georgi), for a tricky reason: Python treats whatever is returned from a __unicode__ method as a unicode object. So it subsequently was trying to re-encode the result into a string, rather than using the string object we were returning. Forcing the method to return a pure unicode object cures the problem.

[Malcolm Tredinnick]

(In [4924]) Fixed #3597 -- Fixed unicode encoding problem in form rendering. Thanks,
Georgi Stanojevski and Ville S?\195?\164?\195?\164vuori.