id,summary,reporter,owner,description,type,status,component,version,severity,resolution,keywords,cc,stage,has_patch,needs_docs,needs_tests,needs_better_patch,easy,ui_ux 7806,django.template refactoring,Johannes Dollinger,Aymeric Augustin,"== Why should `django.template` be refactored? == Filter and Variable parsing is inconsistent. Many ad hoc parsers in defaulttags are fragile. Whitespace handling is ungraceful. The patch provided splits `__init__.py` into separate modules and introduces a `TokenStream` class that allows parsing of literals, vars (called lookups) and filter expressions. No docs yet, have a look at the source (it all happens in ~150 loc). The patch only touches `django.template` and `django.templatetags`. Tests are separate so you can run them with the old code. == Modules in `django.template` == * `context` (no dependencies)[[BR]] Untouched. * `expressions` (no dependencies)[[BR]] Contains `Lookup`, `Literal`, and `FilterExpression`. All are subclasses of a marker class `Expression`. These replace `Variable` and the old `FilterExpression` class. A `Variable` equivalent is provided in `compat.Variable`. * `nodes` (depends on expressions)[[BR]] Contains `Node`, `NodeList`, `TextNode`, and `ExpressionNode`. The latter is the old `VariableNode` renamed. A `VariableNode` alias is provided in `compat`. * `compiler` (depends on context, expressions, nodes)[[BR]] Contains `Parser`, `Lexer`, `Token`, `TemplateSyntaxError`, `Origin`, `StringOrgin`, and `compile_string()`. Those are mostly untouched. Additionally there are `TokenStream`, and `TokenSyntaxError`. Those provide a safe way to parse `Tokens` containing expressions. * `library` (depends on compiler, nodes, context)[[BR]] Contains `InvalidTemplateLibrary`, `Library`, `get_library()`, and `add_to_builtins()`. Mostly untouched. Should be called `libraries` for consistency, but that would clash with the legacy dict. * `loader`[[BR]] Untouched. Moved `TemplateDoesNotExist` here. * `utils`[[BR]] `ConditionalNode`, base class for `IfNode`, `IfEqualNode`. `EmptyNode`. Helper functions: `parse_conditional_nodelists(parser, name)`, `parse_as(bits)` `parse_args_and_kwargs(bits)`, `resolve_args_and_kwargs(args, kwargs)` * `compat`[[BR]] Provides backwards compatibility for `Variable`, `VariableNode`, `TokenParser`, and `resolve_variable()` * `defaulttags`, `loader_tags`[[BR]] Mostly refactored to use `TokenStream` where appropriate. == Tickets == * #4746 - allow whitespace before and after filter separator[[BR]] fixed. tests are there (filter-syntax03, filter-syntax04) but expect TemplateSyntaxError. * #5270 - empty string literals[[BR]] fixed. tests included. * #5756 - accept filters (almost) everywhere[[BR]] fixed. a few tests included. * #5862 - dup of #5756[[BR]] fixed. * #5971 - token parser bug[[BR]] `TokenParser` will be deprecated. * #6271 - filter arguments with spaces[[BR]] fixed. test included. (seems to be fixed in trunk already) * #6296 - expressions for ifequal[[BR]] fixed. tests included (dup of #5756 ?) * #6510 - `get_nodes_by_type()` problem[[BR]] probably fixed. * #6535 - negative numeric literals[[BR]] fixed. tests included. * #7295 - quotes, escaping and translation of string literals handled inconsistently in templates[[BR]] fixed. == Performance == Comparing `runtests.py templates` measures, the refactored version is a bit slower. == Todo == `{% if not %}` is currently valid. The refactored implementation is greedy and reports a syntax error. Could be fixed, not sure it's worth it. Broken tests are: if-tag-not02, if-tag-not03. DDN. Numeric literals ending with ""."" are now considered a syntax error. Broken test: ifequal-numeric07. DDN. `{% url %}` currently accepts an arbitrary unquoted unicode string for the view name. The refactored implementation only accepts bare python identifiers or quoted string literals. Broken test: url05. DDN. `TemplateSyntaxError` messages need some work. Eventually shortcut methods on `TokenStream`. Find better names for `TokenSyntaxError` and `TokenStream`. Provide a decorator that handles the boilerplate `bits=parser.token_stream(token)` and `bits.assert_consumed()` calls. Investigate performance impact. Docs and more tests. ",Cleanup/optimization,closed,Template system,dev,Normal,fixed,tplrf,Carl Meyer research@… Alexander Koshelev,Accepted,1,1,1,1,0,0