[patch] Auto-escaping template changes

Core changes and test suite

The mailing list thread for discussing this patch is here. Please use that instead of this ticket for the time being so that the conversation does not happen in two places at once.

Good job on the patch, Malcom :)

A couple of points: If a markup filter fails due to an import error, I don't think it should be marked as safe. From a skim read, I'm missing the purpose of having an .is_safe property on filters - can't you just check the outputted string to see if it's SafeData??

Update core changes (supercedes previous patch)

Changes to most of the contrib/ applications (excludes admin)

Changes to contrib/admin/

Adrian, this one's back in your court now if you want to give it a review. I think it's complete.

The patches are all independent of each other (each file is only changed by one patch), although 01-core-changes.diff should be applied first. It incorporates all our discussions from OSCON:

1. off by default
2. inherits through template extension (so you can mark the "content" in the base template with "autoescape" and be done with it).
3. adds both "autoescape" and "noautoescape" as per Jacob's suggestion (so that the end tags tell you which tag you are closing explicitly).

The admin change is probably the most destabilising, since mistakes there will be noticed by everybody. It's also probably the hardest one to test in all its variations. I've used it a bit and tried to find problems, but ultimately we are just going to have to be responsive to feedback and fix things as they are noticed. Very few backwards-incompatible changes in the end. The "escape" filter only applying once is the only real one (and we have the new "force_escape" filter if multi-escaping is needed).

Hmm .. Trac's diff colorizer doesn't like new files being added in the diff. A couple of files appears as being patches to /dev/null. They are the addition of django/utils/safestring.py and tests/othertests/autoescape.py respectively, both in 01-core-changes.diff

The raw diff file has the correct information and can be applied with patch -p1 -i ....

Lest it fly under the Adrian radar, I just want to add that my alternative solution is nearly entirely self-contained, so would require no changes at all to the rest of django if it were to be implemented.

I've been using it for a project (including inside of the existing Admin site) with no hazardous side effects.

#2984 was a duplicate.

Malcolm stated that this is still work in progress.

updated patch

updated patch

updated patch

I've brought the patches up to date. Bigger changes were:

• Adapting the tests, this needed some changes in the templates tests. The autoescape tests subclass the templates tests, and this needed a few hooks (but should be ovious)

• SafeString? needs a str() method, or else the one from the normal python string is used, and then it's not SafeData? anymore.

It still needs patches for newforms, but this is a fast moving target if I've ever seen one ...

don't mark unprocessed markup as safe; fix a string initializer

updated patch, includes part of smurf's patch, for rev. 4659

updated patch, includes part of smurf's patch, for rev. 4659

I've brought the first two patches up to date for revision 3659. They contain smurf's patch, and 01-core also adds empty init.py and models.py files who seemed to have slipped somehow.

It doesn't make sense for me to maintain 03-admin since I don't use the admin anywhere.

patch 01-core-changes: the template tag filter needs to be able to deal with functions decorated by stringfilter (since you cannot use escape or safe with filter).

updated patch, see comment

In the current patch, the example in documentation of is_safe isn't actually safe - the value passed could be a "safe" string which contains a unicode HTML attribute 'į' so this filter will introduce a dangerous character ('&').

 	707	 2. If your filter is given a "safe" string, is it guaranteed to return a 
 	708	 "safe" string? If so, set the ``is_safe`` attribute on the function to be 
 	709	 ``True``. For example, a filter that replaced all numbers with the number 
 	710	 spelt out in words is going to be safe-string-preserving, since it cannot 
 	711	 introduce any of the five dangerous characters: <, >, ", ' or &. So we can 
 	712	 write:: 
 	713	 
 	714	    @register.filter 
 	715	    def convert_to_words(value): 
 	716	        # ... implementation here ... 
 	717	        return result 
 	718	 
 	719	    convert_to_words.is_safe = True 

A small bug: Tracebacks for unexpected exceptions in template tags ("Template Error") show a part of the template where the bug hit. With autoescape on, the lines in the template display are escaped one time too often.

core changes ported for the unicode branch

core changes ported for the unicode branch (trac-readable)

contrib changes ported for the unicode branch

core changes ported for the unicode branch (for trac, 3rd attempt)

I have attached two patches ported to the unicode branch; these positively need further review. Somehow trac refuses to display unicode-autoescape-1, I have no idea why. Sorry for spoiling the history ...

updated patch for svn release 5722

updated patch for svn release 5722 (2nd attempt ...)

updated patch for svn release 5722 (3rd attempt)

updated patch for svn release 5722 (4th attempt)

updated patch for svn release 5722

I brought the first two patches up to date to release 5722. I am aware that trac doesn't parse rev5722-01-core-changes.*.diff, but as much as I tried, I was unable to correct it. Please use rev5722-01-core-changes.3.diff and rev5722-02-misc-contrib-changes.diff

updated patch

updated patch

VariableNode.render() must call force_unicode earlier, so that it can see whether it needs to escape or not. I'll add a patch for this separately (variable-node.diff)

BTW: The patches do not (yet) deal with newforms.

see comment 17

(In [6671]) Implemented auto-escaping of variable output in templates. Fully controllable by template authors and it's possible to write filters and templates that simulataneously work in both auto-escaped and non-auto-escaped environments if you need to. Fixed #2359

See documentation in templates.txt and templates_python.txt for how everything works.

Backwards incompatible if you're inserting raw HTML output via template variables.

Based on an original design from Simon Willison and with debugging help from Michael Radziej.