USE_THOUSAND_SEPARATOR fails with UnicodeDecodeError in several locales

[Marti Raudsepp]

When I'm trying to use the {{ foo|floatformat }} template tag with USE_THOUSAND_SEPARATOR=True and numbers >= 1000, it fails with UnicodeDecodeError, resulting in an empty value in the rendered template. This is a pretty plain Ubuntu 10.10 machine, tested with an empty Django project on Ubuntu-packaged Django version 1.2.3 as well as SVN trunk.

Apparently this happens because THOUSANDS_SEPARATOR in django.conf.locale.et.formats is a normal str object -- not Unicode -- but contains an UTF-8 no-break space sequence.

It seems that the same no-break string is also used in bg, fi, hu, lv and uk locales!

% django-admin startproject foo
% cd foo
% ./manage.py shell
Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) 
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: from django.utils import translation

In [2]: from django.template.defaultfilters import floatformat

In [3]: from django.conf import settings

In [4]: settings.NUMBER_GROUPING=3

In [5]: settings.USE_THOUSAND_SEPARATOR=True

In [6]: translation.get_language()
Out[6]: 'en-us'

In [7]: floatformat(1000)
Out[7]: u'1,000'

In [8]: translation.activate('et')

In [9]: translation.get_language()
Out[9]: 'et'

In [10]: floatformat(1000)
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)

/tmp/foo/<ipython console> in <module>()

/usr/lib/pymodules/python2.6/django/template/defaultfilters.pyc in floatformat(text, arg)
    164 
    165     if not m and p < 0:
--> 166         return mark_safe(formats.number_format(u'%d' % (int(d)), 0))
    167 
    168     if p == 0:

/usr/lib/pymodules/python2.6/django/utils/formats.pyc in number_format(value, decimal_pos)
     72         decimal_pos,
     73         get_format('NUMBER_GROUPING'),
---> 74         get_format('THOUSAND_SEPARATOR'),
     75     )
     76 

/usr/lib/pymodules/python2.6/django/utils/numberformat.pyc in format(number, decimal_sep, decimal_pos, grouping, thousand_sep)
     35         for cnt, digit in enumerate(int_part[::-1]):
     36             if cnt and not cnt % grouping:
---> 37                 int_part_gd += thousand_sep
     38             int_part_gd += digit
     39         int_part = int_part_gd[::-1]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

In [11]: from django.utils import formats

In [12]: formats.get_format('THOUSAND_SEPARATOR')
Out[12]: '\xc2\xa0'

In [13]: formats.get_format_modules()
Out[13]: [<module 'django.conf.locale.et.formats' from '/usr/lib/pymodules/python2.6/django/conf/locale/et/formats.pyc'>]

[Claude Paroz]

Unicode strings in Python files should be prefixed with 'u'.

[Claude Paroz]

Add 'u' before unicode strings

[Claude Paroz]

We could even add the u before the strings in the en/formats.py to serve as good model.

[Marti Raudsepp]

I'd suggest using something like u'\u00a0', that way it's clear that it's not just a regular space, but a Unicode character.

[Claude Paroz]

I don't like Unicode sequences exposed to users. I'm adding a second version of the patch with comments instead.

[Claude Paroz]

Also add comments (and en/formats.py)

[Jannis Leidel]

(In [14708]) Fixed #14721 -- Made the THOUSAND_SEPERATOR a unicode string in a few locales. Thanks, Claude Paroz.

[Jannis Leidel]

(In [14712]) [1.2.X] Fixed #14721 -- Made the THOUSAND_SEPERATOR a unicode string in a few locales. Thanks, Claude Paroz.

Backport from trunk (r14708).

[Jacob]

Milestone 1.3 deleted