Django Template Language doesn't support all float literals

[Lily Foote]

When parsing arguments to filters, Django converts numeric literals to float or int, but in the case of scientific notation for float this is incomplete:

>>> from django.template.base import Template
>>> from django.template.engine import Engine
>>> 5.2e3
5200.0
>>> Template("{{ foo|default:5.2e3 }}", engine=e)
<Template template_string="{{ foo|default:5.2e3...">
>>> 5.2e-3
0.0052
>>> Template("{{ foo|default:5.2e-3 }}", engine=e)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-templates/3.12/lib/python3.12/site-packages/django/template/base.py", line 154, in __init__
    self.nodelist = self.compile_nodelist()
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-templates/3.12/lib/python3.12/site-packages/django/template/base.py", line 196, in compile_nodelist
    nodelist = parser.parse()
               ^^^^^^^^^^^^^^
  File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-templates/3.12/lib/python3.12/site-packages/django/template/base.py", line 489, in parse
    raise self.error(token, e)
  File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-templates/3.12/lib/python3.12/site-packages/django/template/base.py", line 487, in parse
    filter_expression = self.compile_filter(token.contents)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-templates/3.12/lib/python3.12/site-packages/django/template/base.py", line 605, in compile_filter
    return FilterExpression(token, self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-templates/3.12/lib/python3.12/site-packages/django/template/base.py", line 706, in __init__
    raise TemplateSyntaxError(
django.template.exceptions.TemplateSyntaxError: Could not parse the remainder: '-3' from 'foo|default:5.2e-3'

I couldn't actually find the documentation for Django's handling of numeric literals in templates, so this may be an undocumented implementation detail. If so, perhaps deprecating scientific notation would be appropriate. Otherwise, we should support this case.

[Aryaman]

I am beginner in os , Giving this a try

[Sarah Boyce]

Thank you! We try to support scientific numbers so we should fix this.
I believe their is an issue with the regex, this might work

django/template/base.py

@@ -640 +640 @@
          )?
  )""" % {
     "constant": constant_string,
-    "num": r"[-+.]?\d[\d.e]*",
+    "num": r"[-+.]?\d[\d.e-]*",
     "var_chars": r"\w\.",
     "filter_sep": re.escape(FILTER_SEPARATOR),

Will need a test

[IronJam]

Yeah it looks like a regex error. I can code a regression test and make a pr.

[IronJam]

PR: - ​https://github.com/django/django/pull/19146

[Hailey Johnson]

​https://github.com/django/django/pull/19213

[GitHub <noreply@…>]

In 5183f7c2:

Fixed #35816 -- Handled parsing of scientific notation in DTL. (#19213)

• Refs #35816 -- Improved test coverage of FilterExpression.

• Fixed #35816 -- Made FilterExpression parse scientific numbers.

---

Co-authored-by: Sarah Boyce <42296566+sarahboyce@…>

[Sarah Boyce]

Note that despite selecting "rebase and merge" on the PR, this appears to be squashed merged. I have now disabled the squash merge on the project.

[Jon Banafato]

I have a question about template behavior that was changed by this feature (please let me know if I should post this as a new ticket instead of continuing conversation here). While ​investigating CI test failures for django-hosts, I observed that Django no longer throws a TemplateSyntaxError for template variable names containing dashes and instead appears to treat them as regular template variable names. I did not see any notes in the ​template language documentation or the templates section of the ​release notes for 6.0. I believe this change was introduced by the commit associated with this issue, and I've included an example of the behavioral change below.

On commit 8df5ce80d26824ce72af41edc03275d435de9432, the following code fails with a TemplateSyntaxError as expected.

>>> from django.template import Template, Context
>>> template = Template("Hello, {{ first-name }}")
Traceback (most recent call last):
...
django.template.exceptions.TemplateSyntaxError: Could not parse the remainder: '-name' from 'first-name'

On commit 5183f7c287a9a5d61ca1103b55166cda5, the following code parses the template and renders the first-name provided.

>>> from django.template import Template, Context
>>> template = Template("Hello, {{ first-name }}")
>>> template.render(Context({"first-name": "Jon"}))
'Hello, Jon'

Does this warrant a new ticket for a behavior and / or documentation change? I'd be happy to help where possible, but I'm not very familiar with this change or the Django development process in general, so I am looking for guidance. Thanks!

[Hailey Johnson]

Hi, you're right that this ticket did change that behavior!

I'm happy to add documentation if we decide to keep it as is, I would argue that if we want to revert the behavior, it shouldn't be handled where it was before because we don't differentiate from variable names and numeric values at that point.

[Jon Banafato]

Thanks for the quick response! The template variable name behavior seems like a rather impactful change compared to the state scope of supporting float literals. I wanted to comment here mostly so that the change didn't go unintended and unnoticed.

[Hailey Johnson]

That makes sense! I'd be glad to follow-up either way. I think the '-' character was neither explicitly included or excluded in valid variable naming (unlike, e.g. whitespace characters are explicitly prohibiting and underscores are explicitly allowed).

[Baptiste Mispelon]

Thanks for flagging this Jon! It's always great when bugs get noticed ahead of a release 😅

This looks like a bug to me, so I've opened #36465 (which I marked as a release blocker since it's a breaking change in behavior).

[Sarah Boyce <42296566+sarahboyce@…>]

In 22506b2:

Fixed #36465, Refs #35816 -- Disallowed '+' and '-' characters in template variable names.

Regression in 5183f7c287a9a5d61ca1103b55166cda52d9c647.

Thank you to Jon Banafato and Baptiste Mispelon for the report.