id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
14721	USE_THOUSAND_SEPARATOR fails with UnicodeDecodeError in several locales	Marti Raudsepp	Jannis Leidel	"When I'm trying to use the `{{ foo|floatformat }}` template tag with `USE_THOUSAND_SEPARATOR=True` and numbers >= 1000, it fails with `UnicodeDecodeError`, resulting in an empty value in the rendered template. This is a pretty plain Ubuntu 10.10 machine, tested with an empty Django project on Ubuntu-packaged Django version 1.2.3 as well as SVN trunk.

Apparently this happens because `THOUSANDS_SEPARATOR` in `django.conf.locale.et.formats` is a normal `str` object -- not Unicode -- but contains an UTF-8 no-break space sequence.

It seems that the same no-break string is also used in bg, fi, hu, lv and uk locales!

{{{
% django-admin startproject foo
% cd foo
% ./manage.py shell
Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) 
Type ""copyright"", ""credits"" or ""license"" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: from django.utils import translation

In [2]: from django.template.defaultfilters import floatformat

In [3]: from django.conf import settings

In [4]: settings.NUMBER_GROUPING=3

In [5]: settings.USE_THOUSAND_SEPARATOR=True

In [6]: translation.get_language()
Out[6]: 'en-us'

In [7]: floatformat(1000)
Out[7]: u'1,000'

In [8]: translation.activate('et')

In [9]: translation.get_language()
Out[9]: 'et'

In [10]: floatformat(1000)
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)

/tmp/foo/<ipython console> in <module>()

/usr/lib/pymodules/python2.6/django/template/defaultfilters.pyc in floatformat(text, arg)
    164 
    165     if not m and p < 0:
--> 166         return mark_safe(formats.number_format(u'%d' % (int(d)), 0))
    167 
    168     if p == 0:

/usr/lib/pymodules/python2.6/django/utils/formats.pyc in number_format(value, decimal_pos)
     72         decimal_pos,
     73         get_format('NUMBER_GROUPING'),
---> 74         get_format('THOUSAND_SEPARATOR'),
     75     )
     76 

/usr/lib/pymodules/python2.6/django/utils/numberformat.pyc in format(number, decimal_sep, decimal_pos, grouping, thousand_sep)
     35         for cnt, digit in enumerate(int_part[::-1]):
     36             if cnt and not cnt % grouping:
---> 37                 int_part_gd += thousand_sep
     38             int_part_gd += digit
     39         int_part = int_part_gd[::-1]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

In [11]: from django.utils import formats

In [12]: formats.get_format('THOUSAND_SEPARATOR')
Out[12]: '\xc2\xa0'

In [13]: formats.get_format_modules()
Out[13]: [<module 'django.conf.locale.et.formats' from '/usr/lib/pymodules/python2.6/django/conf/locale/et/formats.pyc'>]

}}}
"		closed	Internationalization	1.2		fixed			Accepted	1	0	1	0	0	0