django.utils.numberformat should be UTF-8 safe

[Jacob Rief]

By adding

from __future__ import unicode_literals

to django/utils/numberformat.py, one could write formatters for numbers in charsets other than ASCII, for instance code block U+2150 through U+218F, which contains the representation of Roman numbers in UTF-8.

[Claude Paroz]

Patch with tests welcome!

[Jacob Rief]

Most other files in folder django/utils.py are UTF-8 safe. Therefore I assume that this line simply has been forgotten - unless there is another good reason.

Before writing a unit test - I don't know where to start from - I'd first like to ask, if this patch will be accepted.

[Claude Paroz]

I'm sure it will. As for tests, see tests/utils_tests/test_numberformat.py.

[alainivars]

I try to work on this ticket but I missing information about the actual problem. I try to impose unicode string as number, decimal_sep and thousand_sep but none of them failed. Can you give additional information what you are trying to accomplish.

[Jacob Rief]

My concrete problem is an implementation of a class derived from Decimal: ​https://github.com/jrief/django-shop/blob/0.3.0.dev/shop/money/money_maker.py

This class overrides the __format__ method, so that Django's numberformat can render instances of that type in the current locale. But my class also uses unicode symbols, for instance the € symbol. Therefore strings in numberformat shall be mixable with unicode strings from other modules.

You can also see it from another perspective.
Most Python files in Django contain this line:

from __future__ import unicode_literals

without that line, unicode literals only work with Python-3. By not adding this, Django intentionally behaves differently in Py-2 and Py-3.

[Claude Paroz]

It would still be nice to have some code with an error traceback to be able to reproduce the failure and build a test.

[Jacob Rief]

Well, without making numberformat.py UTF-8 safe, I get this error:

    str_number = '{:f}'.format(number)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)

in ​https://github.com/django/django/blob/stable/1.8.x/django/utils/numberformat.py#L28

You can emulate this by implementing a special Decimal class, say

# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from decimal import Decimal

class EuroDecimal(Decimal):
    """
    Wrapper for Decimal which prefixes each amount with the € symbol.
    """
    def __format__(self, specifier, context=None, _localeconv=None):
        amount = Decimal.__format__(self, specifier, context, _localeconv)
        return '€ {}'.format(amount)

# print amount in Euros using the current locale
from django.utils.formats import number_format

price = EuroDecimal('1.23')
print number_format(price)

[Claude Paroz]

​https://github.com/django/django/pull/4285

[Claude Paroz <claude@…>]

In df193b3cefda982dfcd4813bf4e2ac09b1358f21:

Fixed #24382 -- Allowed unicode chars inside formatted numbers

Thanks Jacob Rief for the report and Tim Graham for the review.