id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
7806	django.template refactoring	Johannes Dollinger	Aymeric Augustin	"== Why should `django.template` be refactored? ==
Filter and Variable parsing is inconsistent. Many ad hoc parsers in defaulttags are fragile.
Whitespace handling is ungraceful. 

The patch provided splits `__init__.py` into separate modules and introduces a `TokenStream`
class that allows parsing of literals, vars (called lookups) and filter expressions. No docs yet,
have a look at the source (it all happens in ~150 loc).

The patch only touches `django.template` and `django.templatetags`.
Tests are separate so you can run them with the old code.

== Modules in `django.template` ==
    * `context` (no dependencies)[[BR]]
      Untouched.
          
    * `expressions` (no dependencies)[[BR]]
      Contains `Lookup`, `Literal`, and `FilterExpression`. All are subclasses of a marker
      class `Expression`. These replace `Variable` and the old `FilterExpression` class.
      A `Variable` equivalent is provided in `compat.Variable`.
      
    * `nodes` (depends on expressions)[[BR]]
      Contains `Node`, `NodeList`, `TextNode`, and `ExpressionNode`. The latter 
      is the old `VariableNode` renamed. A `VariableNode` alias is provided in `compat`.
    
    * `compiler` (depends on context, expressions, nodes)[[BR]]
      Contains `Parser`, `Lexer`, `Token`, `TemplateSyntaxError`, 
      `Origin`, `StringOrgin`, and `compile_string()`. Those are mostly untouched.
      Additionally there are `TokenStream`, and `TokenSyntaxError`. Those provide 
      a safe way to parse `Tokens` containing expressions.
    
    * `library` (depends on compiler, nodes, context)[[BR]]
      Contains `InvalidTemplateLibrary`, `Library`, `get_library()`, and `add_to_builtins()`.
      Mostly untouched. Should be called `libraries` for consistency, but that would clash 
      with the legacy dict.
      
    * `loader`[[BR]]
      Untouched. Moved `TemplateDoesNotExist` here.
    
    * `utils`[[BR]]
      `ConditionalNode`, base class for `IfNode`, `IfEqualNode`. `EmptyNode`.
      Helper functions:
      `parse_conditional_nodelists(parser, name)`, `parse_as(bits)`
      `parse_args_and_kwargs(bits)`, `resolve_args_and_kwargs(args, kwargs)`
      
    
    * `compat`[[BR]]
      Provides backwards compatibility for `Variable`, `VariableNode`, `TokenParser`,
      and `resolve_variable()`
    
    * `defaulttags`, `loader_tags`[[BR]]
      Mostly refactored to use `TokenStream` where appropriate.
    
== Tickets ==
    * #4746 - allow whitespace before and after filter separator[[BR]]
      fixed. tests are there (filter-syntax03, filter-syntax04) but expect TemplateSyntaxError.
    
    * #5270 - empty string literals[[BR]]
      fixed. tests included.
    
    * #5756 - accept filters (almost) everywhere[[BR]]
      fixed. a few tests included.
    
    * #5862 - dup of #5756[[BR]]
      fixed.
    
    * #5971 - token parser bug[[BR]]
      `TokenParser` will be deprecated.
    
    * #6271 - filter arguments with spaces[[BR]]
      fixed. test included. (seems to be fixed in trunk already)
    
    * #6296 - expressions for ifequal[[BR]]
      fixed. tests included (dup of #5756 ?)
    
    * #6510 - `get_nodes_by_type()` problem[[BR]]
      probably fixed.
    
    * #6535 - negative numeric literals[[BR]]
      fixed. tests included.
      
    * #7295 - quotes, escaping and translation of string literals handled inconsistently in templates[[BR]]
      fixed.
    
== Performance ==
Comparing `runtests.py templates` measures, the refactored version is a bit slower.

== Todo ==
`{% if not %}` is currently valid. The refactored implementation is greedy and reports a syntax error.
Could be fixed, not sure it's worth it. Broken tests are: if-tag-not02, if-tag-not03. DDN.

Numeric literals ending with ""."" are now considered a syntax error.
Broken test: ifequal-numeric07. DDN.

`{% url %}` currently accepts an arbitrary unquoted unicode string for the view name.
The refactored implementation only accepts bare python identifiers or quoted string literals.
Broken test: url05. DDN.

`TemplateSyntaxError` messages need some work. Eventually shortcut methods on `TokenStream`.

Find better names for `TokenSyntaxError` and `TokenStream`.

Provide a decorator that handles the boilerplate `bits=parser.token_stream(token)`
and `bits.assert_consumed()` calls.

Investigate performance impact.

Docs and more tests.
"	Cleanup/optimization	closed	Template system	dev	Normal	fixed	tplrf	Carl Meyer research@… Alexander Koshelev	Accepted	1	1	1	1	0	0