Code

Opened 8 years ago

Closed 6 years ago

#2359 closed enhancement (fixed)

[patch] Auto-escaping template changes

Reported by: mtredinnick Owned by: nobody
Component: Template system Version:
Severity: normal Keywords:
Cc: mir@… Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

This is an initial patch implementing auto-escaping support. It follows the Simon Willison's AutoEscaping proposal fairly closely. Putting this up now to aid discussion about some of the uncertainties.

Still to come:

  • porting the contrib/ apps over to this
  • documentation patch.

Attachments (23)

autoescape.diff (44.4 KB) - added by mtredinnick 8 years ago.
Core changes and test suite
01-core-changes.diff (60.8 KB) - added by mtredinnick 8 years ago.
Update core changes (supercedes previous patch)
02-misc-contrib-changes.diff (11.5 KB) - added by mtredinnick 8 years ago.
Changes to most of the contrib/ applications (excludes admin)
03-admin-changes.diff (36.9 KB) - added by mtredinnick 8 years ago.
Changes to contrib/admin/
01-core-changes.2.diff (69.7 KB) - added by Michael Radziej <mir@…> 7 years ago.
updated patch
02-misc-contrib-changes.2.diff (12.0 KB) - added by Michael Radziej <mir@…> 7 years ago.
updated patch
03-admin-changes.2.diff (30.9 KB) - added by Michael Radziej <mir@…> 7 years ago.
updated patch
auto.patch (2.0 KB) - added by smurf@… 7 years ago.
don't mark unprocessed markup as safe; fix a string initializer
01-core-changes.3.diff (70.9 KB) - added by Michael Radziej <mir@…> 7 years ago.
updated patch, includes part of smurf's patch, for rev. 4659
02-misc-contrib-changes.3.diff (11.5 KB) - added by Michael Radziej <mir@…> 7 years ago.
updated patch, includes part of smurf's patch, for rev. 4659
01-core-changes.4.diff (71.0 KB) - added by Michael Radziej <mir@…> 7 years ago.
updated patch, see comment
unicode-autoescape-1.diff (73.6 KB) - added by mir@… 7 years ago.
core changes ported for the unicode branch
unicode-autoescape-1.2.diff (71.9 KB) - added by mir@… 7 years ago.
core changes ported for the unicode branch (trac-readable)
unicode-autoescape-2.diff (11.2 KB) - added by mir@… 7 years ago.
contrib changes ported for the unicode branch
unicode-autoescape-1.3.diff (71.9 KB) - added by mir@… 7 years ago.
core changes ported for the unicode branch (for trac, 3rd attempt)
autoescape-1.diff (73.0 KB) - added by mir@… 7 years ago.
updated patch for svn release 5722
rev5722-01-core-changes.diff (71.2 KB) - added by mir@… 7 years ago.
updated patch for svn release 5722 (2nd attempt ...)
rev5722-01-core-changes.2.diff (69.7 KB) - added by mir@… 7 years ago.
updated patch for svn release 5722 (3rd attempt)
rev5722-01-core-changes.3.diff (73.0 KB) - added by mir@… 7 years ago.
updated patch for svn release 5722 (4th attempt)
rev5722-02-misc-contrib-changes.diff (11.9 KB) - added by mir@… 7 years ago.
updated patch for svn release 5722
rev5774-01-core-changes.diff (73.1 KB) - added by mir@… 7 years ago.
updated patch
rev5774-02-misc-contrib-changes.diff (11.9 KB) - added by mir@… 7 years ago.
updated patch
variable-node.diff (991 bytes) - added by mir@… 7 years ago.
see comment 17

Download all attachments as: .zip

Change History (41)

Changed 8 years ago by mtredinnick

Core changes and test suite

comment:1 Changed 8 years ago by mtredinnick

The mailing list thread for discussing this patch is here. Please use that instead of this ticket for the time being so that the conversation does not happen in two places at once.

comment:2 Changed 8 years ago by SmileyChris

Good job on the patch, Malcom :)

A couple of points:
If a markup filter fails due to an import error, I don't think it should be marked as safe.
From a skim read, I'm missing the purpose of having an .is_safe property on filters - can't you just check the outputted string to see if it's SafeData?

comment:3 Changed 8 years ago by mtredinnick

  • Owner changed from adrian to mtredinnick

Changed 8 years ago by mtredinnick

Update core changes (supercedes previous patch)

Changed 8 years ago by mtredinnick

Changes to most of the contrib/ applications (excludes admin)

Changed 8 years ago by mtredinnick

Changes to contrib/admin/

comment:4 Changed 8 years ago by mtredinnick

  • Owner changed from mtredinnick to adrian

Adrian, this one's back in your court now if you want to give it a review. I think it's complete.

The patches are all independent of each other (each file is only changed by one patch), although 01-core-changes.diff should be applied first. It incorporates all our discussions from OSCON:

  1. off by default
  2. inherits through template extension (so you can mark the "content" in the base template with "autoescape" and be done with it).
  3. adds both "autoescape" and "noautoescape" as per Jacob's suggestion (so that the end tags tell you which tag you are closing explicitly).

The admin change is probably the most destabilising, since mistakes there will be noticed by everybody. It's also probably the hardest one to test in all its variations. I've used it a bit and tried to find problems, but ultimately we are just going to have to be responsive to feedback and fix things as they are noticed. Very few backwards-incompatible changes in the end. The "escape" filter only applying once is the only real one (and we have the new "force_escape" filter if multi-escaping is needed).

comment:5 Changed 8 years ago by mtredinnick

Hmm .. Trac's diff colorizer doesn't like new files being added in the diff. A couple of files appears as being patches to /dev/null. They are the addition of django/utils/safestring.py and tests/othertests/autoescape.py respectively, both in 01-core-changes.diff

The raw diff file has the correct information and can be applied with patch -p1 -i ....

comment:6 Changed 8 years ago by SmileyChris

Lest it fly under the Adrian radar, I just want to add that my alternative solution is nearly entirely self-contained, so would require no changes at all to the rest of django if it were to be implemented.

I've been using it for a project (including inside of the existing Admin site) with no hazardous side effects.

comment:7 Changed 7 years ago by adrian

#2984 was a duplicate.

comment:8 Changed 7 years ago by mir@…

  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Design decision needed

Malcolm stated that this is still work in progress.

Changed 7 years ago by Michael Radziej <mir@…>

updated patch

Changed 7 years ago by Michael Radziej <mir@…>

updated patch

Changed 7 years ago by Michael Radziej <mir@…>

updated patch

comment:9 Changed 7 years ago by Michael Radziej <mir@…>

I've brought the patches up to date. Bigger changes were:

  • Adapting the tests, this needed some changes in the templates tests. The autoescape tests subclass the templates tests, and this needed a few hooks (but should be ovious)
  • SafeString needs a str() method, or else the one from the normal python string is used, and then it's not SafeData anymore.

It still needs patches for newforms, but this is a fast moving target if I've ever seen one ...

Changed 7 years ago by smurf@…

don't mark unprocessed markup as safe; fix a string initializer

Changed 7 years ago by Michael Radziej <mir@…>

updated patch, includes part of smurf's patch, for rev. 4659

Changed 7 years ago by Michael Radziej <mir@…>

updated patch, includes part of smurf's patch, for rev. 4659

comment:10 Changed 7 years ago by Michael Radziej <mir@…>

I've brought the first two patches up to date for revision 3659. They contain smurf's patch, and 01-core also adds empty init.py and models.py files who seemed to have slipped somehow.

It doesn't make sense for me to maintain 03-admin since I don't use the admin anywhere.

comment:11 Changed 7 years ago by Michael Radziej <mir@…>

patch 01-core-changes: the template tag filter needs to be able to deal with functions decorated by stringfilter (since you cannot use escape or safe with filter).

Changed 7 years ago by Michael Radziej <mir@…>

updated patch, see comment

comment:12 Changed 7 years ago by Michael Radziej <mir@…>

  • Cc mir@… added

comment:13 Changed 7 years ago by SmileyChris

In the current patch, the example in documentation of is_safe isn't actually safe - the value passed could be a "safe" string which contains a unicode HTML attribute '&#303;' so this filter will introduce a dangerous character ('&').

 	707	 2. If your filter is given a "safe" string, is it guaranteed to return a 
 	708	 "safe" string? If so, set the ``is_safe`` attribute on the function to be 
 	709	 ``True``. For example, a filter that replaced all numbers with the number 
 	710	 spelt out in words is going to be safe-string-preserving, since it cannot 
 	711	 introduce any of the five dangerous characters: <, >, ", ' or &. So we can 
 	712	 write:: 
 	713	 
 	714	    @register.filter 
 	715	    def convert_to_words(value): 
 	716	        # ... implementation here ... 
 	717	        return result 
 	718	 
 	719	    convert_to_words.is_safe = True 

comment:14 Changed 7 years ago by mir@…

A small bug: Tracebacks for unexpected exceptions in template tags ("Template Error") show a part of the template where the bug hit. With autoescape on, the lines in the template display are escaped one time too often.

Changed 7 years ago by mir@…

core changes ported for the unicode branch

Changed 7 years ago by mir@…

core changes ported for the unicode branch (trac-readable)

Changed 7 years ago by mir@…

contrib changes ported for the unicode branch

Changed 7 years ago by mir@…

core changes ported for the unicode branch (for trac, 3rd attempt)

comment:15 Changed 7 years ago by mir@…

I have attached two patches ported to the unicode branch; these positively need further review. Somehow trac refuses to display unicode-autoescape-1, I have no idea why. Sorry for spoiling the history ...

Changed 7 years ago by mir@…

updated patch for svn release 5722

Changed 7 years ago by mir@…

updated patch for svn release 5722 (2nd attempt ...)

Changed 7 years ago by mir@…

updated patch for svn release 5722 (3rd attempt)

Changed 7 years ago by mir@…

updated patch for svn release 5722 (4th attempt)

Changed 7 years ago by mir@…

updated patch for svn release 5722

comment:16 Changed 7 years ago by mir@…

I brought the first two patches up to date to release 5722. I am aware that trac doesn't parse rev5722-01-core-changes.*.diff, but as much as I tried, I was unable to correct it. Please use rev5722-01-core-changes.3.diff and rev5722-02-misc-contrib-changes.diff

Changed 7 years ago by mir@…

updated patch

Changed 7 years ago by mir@…

updated patch

comment:17 Changed 7 years ago by mir@…

VariableNode.render() must call force_unicode earlier, so that it can see whether it needs to escape or not. I'll add a patch for this separately (variable-node.diff)

BTW: The patches do not (yet) deal with newforms.

Changed 7 years ago by mir@…

see comment 17

comment:18 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [6671]) Implemented auto-escaping of variable output in templates. Fully controllable by template authors and it's possible to write filters and templates that simulataneously work in both auto-escaped and non-auto-escaped environments if you need to. Fixed #2359

See documentation in templates.txt and templates_python.txt for how everything
works.

Backwards incompatible if you're inserting raw HTML output via template variables.

Based on an original design from Simon Willison and with debugging help from Michael Radziej.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.