template "floatformat" tag throws ValueError for big numbers in Python 3

[mateuszf]

I'm porting application from Python 2 to Python 3.
We receive and display floating point values from external system. They may be big values.

Following example works on Python 2 with Django 1.10.2 but throws "ValueError: valid range for prec is [1, MAX_PREC]" on Python 3 with the same Django version.

def my_view(request):
    return render(request, my_template.html', { 'value':-1.323297138040798e+35 })

{{ value|floatformat:2 }}

We have separate custom tag for displaying values in scientific format but how am I supposed to check if value will be printed in scientific format before printing it in template ? This was working automatically in Python 2.

[Tim Graham]

A regression test is attached, though it doesn't work on Python 2 either, returning the original value rather than the formatted version.

[Mariusz Felisiak]

Test

[Mariusz Felisiak]

Replying to Tim Graham:

A regression test is attached, though it doesn't work on Python 2 either, returning the original value rather than the formatted version.

I attached correct regression test. Previous was little bit wrong because

-1.323297138040798e+35 = -1.323297138040798 * 10³⁵ = -132329713804079800000000000000000000

PR in progress.

[Mariusz Felisiak]

​PR

[mateuszf]

I've looked at the PR and the test is:

self.assertEqual(floatformat(-1.323297138040798e+35, 2), '-132329713804079800000000000000000000.00')

But I've just retested it on Python 2 and my example is being rendered as: -1.32329713804e+35. So in Python 2 "floatformat" does not force big numbers to standard notation, and I think this tag should behave the same for Python 2 and Python 3.

From my project's perspective: values we receive from external system may be big or may be small but we always have some user defined precision. If "floatformat" will always force standard notation even for really big numbers it will become useless for us, because we don't want to print values in a way that will occupy 1/2 screen width. It's also unreadable this way.

[Tim Graham]

Right, it's debatable whether or not the behavior on Python 2 should be considered a bug. The current behavior of leaving scientific notation untouched isn't documented. I don't have a strong opinion.

[mateuszf]

I understand that current behavior on Python 2 may be debatable. Maybe there should be 2 versions of "floatformat" with different names, one that will always force standard notation and one that will always print something readable.
We can always implement it manually in our project if the decision will be to leave it as implemented in PR.

[Mariusz Felisiak]

Replying to mateuszf:

I've looked at the PR and the test is:

self.assertEqual(floatformat(-1.323297138040798e+35, 2), '-132329713804079800000000000000000000.00')

But I've just retested it on Python 2 and my example is being rendered as: -1.32329713804e+35. So in Python 2 "floatformat" does not force big numbers to standard notation, and I think this tag should behave the same for Python 2 and Python 3.

Are you sure? This test passed on Python 2 and Python 3, hence in both cases floatformat returns '-132329713804079800000000000000000000.00' ie. number in a standard notation (without losing precision).

[Tim Graham]

The point is that the current behavior on Python 2 leaves numbers in scientific notation untouched. Absent another argument, I don't mind keeping that behavior. I guess there's little reason to break backwards-compatibility and using floatformat with scientific notation to add a bunch of zeros doesn't seem particularly useful.

[mateuszf]

Replying to felixxm:

Are you sure? This test passed on Python 2 and Python 3, hence in both cases floatformat returns '-132329713804079800000000000000000000.00' ie. number in a standard notation (without losing precision).

Well that's strange. My result was pasted directly from browser window running simple test project using Python 2.7.8 (32-bit) on Windows 7 (64-bit).
I've made some more tests reinstalling Python to Python 2.7.12 (32-bit) and Python 2.7.12 (64-bit) using the same test project and the same Django 1.10.2 and result is still the same: -1.32329713804e+35

[Tim Graham]

felixxm, do you agree to keep the current behavior:

>>> floatformat(-1.323297138040798e+35, 2)
u'-1.32329713804e+35'

or do you feel there's value in changing it to output all the zeros (as done by the current PR)?

[Mariusz Felisiak]

Replying to Tim Graham:

felixxm, do you agree to keep the current behavior:

>>> floatformat(-1.323297138040798e+35, 2)
u'-1.32329713804e+35'

or do you feel there's value in changing it to output all the zeros (as done by the current PR)?

It's really hard to say. The current behavior works only in python2. If we want to leave numbers in scientific notation untouched then it should be fixed for python3 and documented (I can prepare PR).

Moreover ​this change should be done, because without it we loose precision in python2: floatformat(-1.323297138040798e+35, 2) returns u'-1.32329713804e+35' instead of u'-1.323297138040798e+35'.

[Tim Graham]

Yes, that's the plan I envisioned.

[Mariusz Felisiak]

I checked current floatformat behavior in different cases:

• python 2:

>>> floatformat(-1.323297138040798e+35, -2)
u'-132329713804000000000000000000000000'
>>> floatformat(-1.323297138040798e+35, 2)
u'-1.32329713804e+35'
>>> floatformat(1.5e-15, 20)
u'0.00000000000000150000'
>>> floatformat(1.5e-15, -20)
u'0.00000000000000150000'

after PR (in current form):

>>> floatformat(-1.323297138040798e+35, -2)
u'-132329713804079800000000000000000000'
>>> floatformat(-1.323297138040798e+35, 2)
u'-132329713804079800000000000000000000.00'
>>> floatformat(1.5e-15, 20)
u'0.00000000000000150000'
>>> floatformat(1.5e-15, -20)
u'0.00000000000000150000'

• python 3:

>>> from django.template.defaultfilters import floatformat
>>> floatformat(-1.323297138040798e+35, -2)
'-132329713804079800000000000000000000'
>>> floatformat(-1.323297138040798e+35, 2)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/xxxyyy/lib/python3.4/site-packages/django/template/defaultfilters.py", line 173, in floatformat
    sign, digits, exponent = d.quantize(exp, ROUND_HALF_UP, Context(prec=prec)).as_tuple()
  File "/usr/lib/python3.4/decimal.py", line 3904, in __init__
    self.prec = prec if prec is not None else dc.prec
  File "/usr/lib/python3.4/decimal.py", line 3957, in __setattr__
    return self._set_integer_check(name, value, 1, 'inf')
  File "/usr/lib/python3.4/decimal.py", line 3938, in _set_integer_check
    raise ValueError("%s must be in [%d, %s]. got: %s" % (name, vmin, vmax, value))
ValueError: prec must be in [1, inf]. got: -1
>>> floatformat(1.5e-15, 20)
'0.00000000000000150000'
>>> floatformat(1.5e-15, -20)
'0.00000000000000150000'

after PR (in current form):

>>> floatformat(-1.323297138040798e+35, -2)
'-132329713804079800000000000000000000'
>>> floatformat(-1.323297138040798e+35, 2)
'-132329713804079800000000000000000000.00'
>>> floatformat(1.5e-15, 20)
'0.00000000000000150000'
>>> floatformat(1.5e-15, -20)
'0.00000000000000150000'

hence maybe it should be done like in the current PR.

[Tim Graham]

Based on those results, I agree the current PR probably makes the most sense.

[Tim Graham <timograham@…>]

In 12f7928:

Fixed #27394 -- Added scientific notation support for big integers in floatformat filter.

[Tim Graham <timograham@…>]

In 41759c90:

Refs #27394 -- Removed @expectedFailure from a floatformat test.