Verbatim tag fails to render curly braces

[stephband]

The {% verbatim %} tag fails to render curly braces under certain conditions - notably when there are two verbatim tags on the same line.

This works:

{% verbatim %}{{{% endverbatim %}   – correctly renders as –   {{
{% verbatim %}}}{% endverbatim %}   – correctly renders as –   }}
{% verbatim %}{ {% endverbatim %}text{% verbatim %} }{% endverbatim %}   – correctly renders as – { text }

However, this fails:

{% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %}   – incorrectly renders as –   {{% endverbatim %}text{% verbatim %}}
{% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %}   – incorrectly renders as –   {{{% endverbatim %}text{% verbatim %}}}
{% verbatim %}{{ {% endverbatim %}text{% verbatim %} }}{% endverbatim %}   – incorrectly renders as –   {{ {% endverbatim %}text{% verbatim %} }}

[Tim Graham]

Not sure if it will be easy/possible to fix. If not, we can document the limitation.

[Paul Egges]

This occurs because the {{ and }} are the starting and ending delimiters for variables. This also affects block tag delimters, {% and %}, and comment tag delimiters, {# and #}. In all cases they must be matching.

[shayneoneill]

Possibly a better example why this is a really huge headache is this

{%verbatim%} {{ {%endverbatim%} computer.{{field}} {%verbatim%} }} {%endverbatim%}

From some auditing software where we are using django views to render lists of fields , which however are being fetched via AngularJS from a third party REST source. We simply can't find a simple way around this. :(

An alternative might even be some simple tags that provide the missing character Ie {% special left_double_brace %} or whatever.

Whilst documentation would at least warn the bug is there, this is a show-stopper error for a lot of use cases.

[Aymeric Augustin]

Shayne, aren't you looking for ​the templatetag tag?

[shayneoneill]

Oh wow. I didn't even realise that existed. Thanks! Still I do think that allowing multiple verbatim/endverbatim instances on a line would solve a lot of headaches.

[Aymeric Augustin]

This may require switching to a proper lexer that understands literals instead of the current regex-based implementation.

[Alexey]

Probably related:

{% verbatim %}{%{% endverbatim %}

With curly bracket and percentage sign (open block tag) template throws exception of unclosed verbatim tag. Django 1.11.22

[Atul Bhouraskar]

I've created a pull request that allows all the examples in this ticket to be correctly rendered - ​https://github.com/django/django/pull/14786
Test cases for the examples have been added and pass. More test cases (for examples not yet reported) may need to be added.

Note that this pull request is based off my proposed fix for #23356 where all verbatim tag logic is isolated into a Parser.parse_verbatim() method and removes all special case logic currently in the Lexer.create_token().

[Mariusz Felisiak]

See comment.

[Chris Jerdonek]

The way I would try solving this today is, after ​PR #14739 is merged, make ​Lexer.create_token() raise a custom exception called something like VerbatimTagStarting with the end string as the argument if a verbatim tag is encountered. (The line where this happens is ​here.) Then, in ​Lexer.tokenize(), handle VerbatimTagStarting being raised by searching for the new end string, and then resuming the tag_re.split() iteration at the new location in the template string afterwards. This will be much easier to implement now that ticket #33002 is resolved. One side benefit of this is that Lexer.create_token() should become simpler (though the complexity will move to tokenize()). However, Lexer.create_token() probably shouldn't be having much logic anyways (nor state, which the changes I'm suggesting would also remove).

[Chris Jerdonek]

Here are a few comments to add to my previous comment, after thinking a little more about this:

I believe that finding the end tag for a custom verbatim tag can be done efficiently and without having to dynamically compile any new regular expressions. (The regex pattern I have in mind here would be one that ends with endverbatim.)

I also think this could be done somewhat more efficiently if Django didn't have to support pathological ​custom closing verbatim tags like this:

{% verbatim {% %}
Avoid template rendering via the {% verbatim %}{% endverbatim %} block.
{% endverbatim {% %}

(In the above, {% is being used as the custom myblock string below.)

{% verbatim myblock %}
Avoid template rendering via the {% verbatim %}{% endverbatim %} block.
{% endverbatim myblock %}

If closing tags like this didn't have to be supported, the approach I have in mind could I think be done with fewer operations since the regex pattern could end with %} instead of endverbatim. (The Python regex module only supports searching for non-overlapping patterns, and whether closing tags like this are allowed affects whether the regex pattern used to search for a closing verbatim tag will be overlapping or not.) But as I said, I believe it can still be done efficiently even if they are allowed.

By the way, would I be allowed to work on this? Atul, I know you assigned yourself, but I'm not sure if that was only on the assumption that this ticket would be resolved using the PR you posted for #23356.

PS - one other thing I noticed is that care needs to be taken to ensure that the regex will match {% endverbatim with any Unicode whitespace between {% and endverbatim (so not just ASCII whitespace), but not \n. That means the regex sequence \s can't be used as is since it includes \n.

[Atul Bhouraskar]

Hi Chris and Mariusz,

Sorry for the delay in responding, was a bit too busy at work.

Let me add in a few general comments about this ticket:

• The {% verbatim %} tags are only found in a very small subset of django templates (I could be very wrong here but I've never actually come across usage out in the wild yet).
• The conditions in this ticket are only found in a small subset of all verbatim usages - ie combined with the above this is an extremely small use case.
• Every example provided here can be implemented with either the templatetag tag as pointed out in https://code.djangoproject.com/ticket/23424#comment:8 or via a custom tag. As older comments have said we could note this in the documentation.
• Adding complexity to create_token() and/or tokenise() - both of which are in the critical path for all template tags (ie. every single template out there is impacted), does not seem (IMHO) a good enough tradeoff to fix this.
• My strong feeling is that any solution to this ticket could always either be considered inefficient or have special cases/ limitations which will need to be spelt out in documentation anyway.
• It would be best to simply acknowledge the current behaviour in the documentation and provide workarounds there - happy to create a PR.

Now coming to my proposed fix (if we are to proceed with the fixing anyway):

• The multiple re-tokenisation only happens if the conditions for this ticket are met (multiple verbatim tags on the same line containing matched var/tag braces within).
• As I've said it is only a proof of concept fix and of course the regex etc definitely can improve.
• The inefficiency will only be encountered if the template being parsed contains a verbatim tag (as parse_verbatim() will never get called otherwise).
• The inefficiency is restricted to the parsing stage which should only ever happen once per template as long as the cached loader is used (which is the default). Users concerned with performance should really not opt out of this.

To summarise:

• I would strongly support leaving the current behaviour as-is and updating the documentation as simple workarounds exist.

PS. On the other hand there is no workaround for creating "verbatim like" tags (#23356) and the proposed fix there has zero impact on all other template tags (and actually reduces complexity in create_token() with a possible small boost in performance).

[Atul Bhouraskar]

Replying to Chris Jerdonek:

By the way, would I be allowed to work on this? Atul, I know you assigned yourself, but I'm not sure if that was only on the assumption that this ticket would be resolved using the PR you posted for #23356.

Sure thing :)
I can de-assign myself if you want.