Inaccurate tags recognition

[tonnzor <tonn81@…>]

Template lexer's check for tags is inaccurate - it checks only the beginning of the tag, not the ending and / or content.

Sample template:

{{ "some text" #}

This will result:

some text

But it should produce:

{{ "some text" #}

[Thomas Steinacher <tom@…>]

Shouldn't your sample give a syntax error, as you're mixing {{ and #}?

[Malcolm Tredinnick]

There is an interesting problem going on here, but it's a little bit bigger than this patch. The patch only works if the broken template is the whole "token_string" contents. Having any extra trailing characters would land you right back in the initial problem. Removing the "has patch" designator for that reason.

The bug itself is only triggered if the very first characters of a string of characters that do not contain a real tag match could be the beginning of a match. Again, putting any random character at the start of the string hides the problem. There is also a problem with typos like

{{ foo #} {{ blah }}

That will try to resolve a variable called "foo #} {{ blah", rather than just spitting out the first bit unchanged and then treating the {{ blah }} portion as a well-formed tag.

I'm working on a slight change to the matching expression that should fix all of these problems without any real performance impact.

Thomas: it's a bit of a coin toss whether this should be treated as a syntax error or not. One problem with doing so is that it would make passing the (somewhat improbable) string "}}" as an argument to a tag problematic. At the moment we display any malformed input like this literally in the resulting template, so it's easy to pick up by eye. I'm not completely unhappy with that behaviour.

[tonnzor <tonn81@…>]

@Thomas Steinacher:

No. This template will be processed:

1. Splitted by tag_re regular expression (result: ['{{ "some text" #}',]
2. Parsed by Lexer.tokenize (result: {{ "some text")
3. Parsed by Lexer.create_token

Code for step 3:

        if token_string.startswith(VARIABLE_TAG_START):
            token = Token(TOKEN_VAR, token_string[len(VARIABLE_TAG_START):-len(VARIABLE_TAG_END)].strip())
        elif token_string.startswith(BLOCK_TAG_START):
            token = Token(TOKEN_BLOCK, token_string[len(BLOCK_TAG_START):-len(BLOCK_TAG_END)].strip())
        elif token_string.startswith(COMMENT_TAG_START):
            token = Token(TOKEN_COMMENT, '')
        else:
            token = Token(TOKEN_TEXT, token_string)

As you see - only the beginning is checked, and tag's end is cut (any 2 chars).

Provided example is edge case, but this bug can cause very strange errors. I can provide move examples if needed.

[tonnzor <tonn81@…>]

@mtredinnick:

Yes, in your example "foo #} {{ blah" is tried to resolve as a variable, but the code related to this resilving should handle it and generate an error.

[Malcolm Tredinnick]

Yes, it does handle that case. It would be nice to less greedy, though. However, may not be possible. Fixing the initial problem is the real goal.

[tonnzor <tonn81@…>]

Before making a patch I thought about better solution, but its implementation takes a lot of code change (update lexer, tags evaluating, etc.) that can cause new bugs, so I made a simple patch that covers the initial defect.

Anyway, the better solution is better ;-)

[Malcolm Tredinnick]

(In [5104]) Fixed #4164, #4171 -- Reworked some of the template lexer logic to ensure we
don't get caught out by a couple of corner cases.