Opened 10 years ago

Closed 9 years ago

#2359 closed enhancement (fixed)

[patch] Auto-escaping template changes

Reported by: Malcolm Tredinnick Owned by: nobody
Component: Template system Version:
Severity: normal Keywords:
Cc: mir@… Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

This is an initial patch implementing auto-escaping support. It follows the Simon Willison's AutoEscaping proposal fairly closely. Putting this up now to aid discussion about some of the uncertainties.

Still to come:

  • porting the contrib/ apps over to this
  • documentation patch.

Attachments (23)

autoescape.diff (44.4 KB) - added by Malcolm Tredinnick 10 years ago.
Core changes and test suite
01-core-changes.diff (60.8 KB) - added by Malcolm Tredinnick 10 years ago.
Update core changes (supercedes previous patch)
02-misc-contrib-changes.diff (11.5 KB) - added by Malcolm Tredinnick 10 years ago.
Changes to most of the contrib/ applications (excludes admin)
03-admin-changes.diff (36.9 KB) - added by Malcolm Tredinnick 10 years ago.
Changes to contrib/admin/
01-core-changes.2.diff (69.7 KB) - added by Michael Radziej <mir@…> 10 years ago.
updated patch
02-misc-contrib-changes.2.diff (12.0 KB) - added by Michael Radziej <mir@…> 10 years ago.
updated patch
03-admin-changes.2.diff (30.9 KB) - added by Michael Radziej <mir@…> 10 years ago.
updated patch
auto.patch (2.0 KB) - added by smurf@… 10 years ago.
don't mark unprocessed markup as safe; fix a string initializer
01-core-changes.3.diff (70.9 KB) - added by Michael Radziej <mir@…> 10 years ago.
updated patch, includes part of smurf's patch, for rev. 4659
02-misc-contrib-changes.3.diff (11.5 KB) - added by Michael Radziej <mir@…> 10 years ago.
updated patch, includes part of smurf's patch, for rev. 4659
01-core-changes.4.diff (71.0 KB) - added by Michael Radziej <mir@…> 10 years ago.
updated patch, see comment
unicode-autoescape-1.diff (73.6 KB) - added by mir@… 9 years ago.
core changes ported for the unicode branch
unicode-autoescape-1.2.diff (71.9 KB) - added by mir@… 9 years ago.
core changes ported for the unicode branch (trac-readable)
unicode-autoescape-2.diff (11.2 KB) - added by mir@… 9 years ago.
contrib changes ported for the unicode branch
unicode-autoescape-1.3.diff (71.9 KB) - added by mir@… 9 years ago.
core changes ported for the unicode branch (for trac, 3rd attempt)
autoescape-1.diff (73.0 KB) - added by mir@… 9 years ago.
updated patch for svn release 5722
rev5722-01-core-changes.diff (71.2 KB) - added by mir@… 9 years ago.
updated patch for svn release 5722 (2nd attempt ...)
rev5722-01-core-changes.2.diff (69.7 KB) - added by mir@… 9 years ago.
updated patch for svn release 5722 (3rd attempt)
rev5722-01-core-changes.3.diff (73.0 KB) - added by mir@… 9 years ago.
updated patch for svn release 5722 (4th attempt)
rev5722-02-misc-contrib-changes.diff (11.9 KB) - added by mir@… 9 years ago.
updated patch for svn release 5722
rev5774-01-core-changes.diff (73.1 KB) - added by mir@… 9 years ago.
updated patch
rev5774-02-misc-contrib-changes.diff (11.9 KB) - added by mir@… 9 years ago.
updated patch
variable-node.diff (991 bytes) - added by mir@… 9 years ago.
see comment 17

Download all attachments as: .zip

Change History (41)

Changed 10 years ago by Malcolm Tredinnick

Attachment: autoescape.diff added

Core changes and test suite

comment:1 Changed 10 years ago by Malcolm Tredinnick

The mailing list thread for discussing this patch is here. Please use that instead of this ticket for the time being so that the conversation does not happen in two places at once.

comment:2 Changed 10 years ago by Chris Beaven

Good job on the patch, Malcom :)

A couple of points:
If a markup filter fails due to an import error, I don't think it should be marked as safe.
From a skim read, I'm missing the purpose of having an .is_safe property on filters - can't you just check the outputted string to see if it's SafeData?

comment:3 Changed 10 years ago by Malcolm Tredinnick

Owner: changed from Adrian Holovaty to Malcolm Tredinnick

Changed 10 years ago by Malcolm Tredinnick

Attachment: 01-core-changes.diff added

Update core changes (supercedes previous patch)

Changed 10 years ago by Malcolm Tredinnick

Changes to most of the contrib/ applications (excludes admin)

Changed 10 years ago by Malcolm Tredinnick

Attachment: 03-admin-changes.diff added

Changes to contrib/admin/

comment:4 Changed 10 years ago by Malcolm Tredinnick

Owner: changed from Malcolm Tredinnick to Adrian Holovaty

Adrian, this one's back in your court now if you want to give it a review. I think it's complete.

The patches are all independent of each other (each file is only changed by one patch), although 01-core-changes.diff should be applied first. It incorporates all our discussions from OSCON:

  1. off by default
  2. inherits through template extension (so you can mark the "content" in the base template with "autoescape" and be done with it).
  3. adds both "autoescape" and "noautoescape" as per Jacob's suggestion (so that the end tags tell you which tag you are closing explicitly).

The admin change is probably the most destabilising, since mistakes there will be noticed by everybody. It's also probably the hardest one to test in all its variations. I've used it a bit and tried to find problems, but ultimately we are just going to have to be responsive to feedback and fix things as they are noticed. Very few backwards-incompatible changes in the end. The "escape" filter only applying once is the only real one (and we have the new "force_escape" filter if multi-escaping is needed).

comment:5 Changed 10 years ago by Malcolm Tredinnick

Hmm .. Trac's diff colorizer doesn't like new files being added in the diff. A couple of files appears as being patches to /dev/null. They are the addition of django/utils/safestring.py and tests/othertests/autoescape.py respectively, both in 01-core-changes.diff

The raw diff file has the correct information and can be applied with patch -p1 -i ....

comment:6 Changed 10 years ago by Chris Beaven

Lest it fly under the Adrian radar, I just want to add that my alternative solution is nearly entirely self-contained, so would require no changes at all to the rest of django if it were to be implemented.

I've been using it for a project (including inside of the existing Admin site) with no hazardous side effects.

comment:7 Changed 10 years ago by Adrian Holovaty

#2984 was a duplicate.

comment:8 Changed 10 years ago by mir@…

Patch needs improvement: set
Triage Stage: UnreviewedDesign decision needed

Malcolm stated that this is still work in progress.

Changed 10 years ago by Michael Radziej <mir@…>

Attachment: 01-core-changes.2.diff added

updated patch

Changed 10 years ago by Michael Radziej <mir@…>

updated patch

Changed 10 years ago by Michael Radziej <mir@…>

Attachment: 03-admin-changes.2.diff added

updated patch

comment:9 Changed 10 years ago by Michael Radziej <mir@…>

I've brought the patches up to date. Bigger changes were:

  • Adapting the tests, this needed some changes in the templates tests. The autoescape tests subclass the templates tests, and this needed a few hooks (but should be ovious)
  • SafeString needs a str() method, or else the one from the normal python string is used, and then it's not SafeData anymore.

It still needs patches for newforms, but this is a fast moving target if I've ever seen one ...

Changed 10 years ago by smurf@…

Attachment: auto.patch added

don't mark unprocessed markup as safe; fix a string initializer

Changed 10 years ago by Michael Radziej <mir@…>

Attachment: 01-core-changes.3.diff added

updated patch, includes part of smurf's patch, for rev. 4659

Changed 10 years ago by Michael Radziej <mir@…>

updated patch, includes part of smurf's patch, for rev. 4659

comment:10 Changed 10 years ago by Michael Radziej <mir@…>

I've brought the first two patches up to date for revision 3659. They contain smurf's patch, and 01-core also adds empty init.py and models.py files who seemed to have slipped somehow.

It doesn't make sense for me to maintain 03-admin since I don't use the admin anywhere.

comment:11 Changed 10 years ago by Michael Radziej <mir@…>

patch 01-core-changes: the template tag filter needs to be able to deal with functions decorated by stringfilter (since you cannot use escape or safe with filter).

Changed 10 years ago by Michael Radziej <mir@…>

Attachment: 01-core-changes.4.diff added

updated patch, see comment

comment:12 Changed 10 years ago by Michael Radziej <mir@…>

Cc: mir@… added

comment:13 Changed 9 years ago by Chris Beaven

In the current patch, the example in documentation of is_safe isn't actually safe - the value passed could be a "safe" string which contains a unicode HTML attribute '&#303;' so this filter will introduce a dangerous character ('&').

 	707	 2. If your filter is given a "safe" string, is it guaranteed to return a 
 	708	 "safe" string? If so, set the ``is_safe`` attribute on the function to be 
 	709	 ``True``. For example, a filter that replaced all numbers with the number 
 	710	 spelt out in words is going to be safe-string-preserving, since it cannot 
 	711	 introduce any of the five dangerous characters: <, >, ", ' or &. So we can 
 	712	 write:: 
 	713	 
 	714	    @register.filter 
 	715	    def convert_to_words(value): 
 	716	        # ... implementation here ... 
 	717	        return result 
 	718	 
 	719	    convert_to_words.is_safe = True 

comment:14 Changed 9 years ago by mir@…

A small bug: Tracebacks for unexpected exceptions in template tags ("Template Error") show a part of the template where the bug hit. With autoescape on, the lines in the template display are escaped one time too often.

Changed 9 years ago by mir@…

Attachment: unicode-autoescape-1.diff added

core changes ported for the unicode branch

Changed 9 years ago by mir@…

Attachment: unicode-autoescape-1.2.diff added

core changes ported for the unicode branch (trac-readable)

Changed 9 years ago by mir@…

Attachment: unicode-autoescape-2.diff added

contrib changes ported for the unicode branch

Changed 9 years ago by mir@…

Attachment: unicode-autoescape-1.3.diff added

core changes ported for the unicode branch (for trac, 3rd attempt)

comment:15 Changed 9 years ago by mir@…

I have attached two patches ported to the unicode branch; these positively need further review. Somehow trac refuses to display unicode-autoescape-1, I have no idea why. Sorry for spoiling the history ...

Changed 9 years ago by mir@…

Attachment: autoescape-1.diff added

updated patch for svn release 5722

Changed 9 years ago by mir@…

updated patch for svn release 5722 (2nd attempt ...)

Changed 9 years ago by mir@…

updated patch for svn release 5722 (3rd attempt)

Changed 9 years ago by mir@…

updated patch for svn release 5722 (4th attempt)

Changed 9 years ago by mir@…

updated patch for svn release 5722

comment:16 Changed 9 years ago by mir@…

I brought the first two patches up to date to release 5722. I am aware that trac doesn't parse rev5722-01-core-changes.*.diff, but as much as I tried, I was unable to correct it. Please use rev5722-01-core-changes.3.diff and rev5722-02-misc-contrib-changes.diff

Changed 9 years ago by mir@…

updated patch

Changed 9 years ago by mir@…

updated patch

comment:17 Changed 9 years ago by mir@…

VariableNode.render() must call force_unicode earlier, so that it can see whether it needs to escape or not. I'll add a patch for this separately (variable-node.diff)

BTW: The patches do not (yet) deal with newforms.

Changed 9 years ago by mir@…

Attachment: variable-node.diff added

see comment 17

comment:18 Changed 9 years ago by Malcolm Tredinnick

Resolution: fixed
Status: newclosed

(In [6671]) Implemented auto-escaping of variable output in templates. Fully controllable by template authors and it's possible to write filters and templates that simulataneously work in both auto-escaped and non-auto-escaped environments if you need to. Fixed #2359

See documentation in templates.txt and templates_python.txt for how everything
works.

Backwards incompatible if you're inserting raw HTML output via template variables.

Based on an original design from Simon Willison and with debugging help from Michael Radziej.

Note: See TracTickets for help on using tickets.
Back to Top