Opened 18 years ago

Closed 16 years ago

#2359 closed enhancement (fixed)

[patch] Auto-escaping template changes

Reported by: Malcolm Tredinnick Owned by: nobody
Component: Template system Version:
Severity: normal Keywords:
Cc: mir@… Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

This is an initial patch implementing auto-escaping support. It follows the Simon Willison's AutoEscaping proposal fairly closely. Putting this up now to aid discussion about some of the uncertainties.

Still to come:

  • porting the contrib/ apps over to this
  • documentation patch.

Attachments (23)

autoescape.diff (44.4 KB ) - added by Malcolm Tredinnick 18 years ago.
Core changes and test suite
01-core-changes.diff (60.8 KB ) - added by Malcolm Tredinnick 18 years ago.
Update core changes (supercedes previous patch)
02-misc-contrib-changes.diff (11.5 KB ) - added by Malcolm Tredinnick 18 years ago.
Changes to most of the contrib/ applications (excludes admin)
03-admin-changes.diff (36.9 KB ) - added by Malcolm Tredinnick 18 years ago.
Changes to contrib/admin/
01-core-changes.2.diff (69.7 KB ) - added by Michael Radziej <mir@…> 17 years ago.
updated patch
02-misc-contrib-changes.2.diff (12.0 KB ) - added by Michael Radziej <mir@…> 17 years ago.
updated patch
03-admin-changes.2.diff (30.9 KB ) - added by Michael Radziej <mir@…> 17 years ago.
updated patch
auto.patch (2.0 KB ) - added by smurf@… 17 years ago.
don't mark unprocessed markup as safe; fix a string initializer
01-core-changes.3.diff (70.9 KB ) - added by Michael Radziej <mir@…> 17 years ago.
updated patch, includes part of smurf's patch, for rev. 4659
02-misc-contrib-changes.3.diff (11.5 KB ) - added by Michael Radziej <mir@…> 17 years ago.
updated patch, includes part of smurf's patch, for rev. 4659
01-core-changes.4.diff (71.0 KB ) - added by Michael Radziej <mir@…> 17 years ago.
updated patch, see comment
unicode-autoescape-1.diff (73.6 KB ) - added by mir@… 17 years ago.
core changes ported for the unicode branch
unicode-autoescape-1.2.diff (71.9 KB ) - added by mir@… 17 years ago.
core changes ported for the unicode branch (trac-readable)
unicode-autoescape-2.diff (11.2 KB ) - added by mir@… 17 years ago.
contrib changes ported for the unicode branch
unicode-autoescape-1.3.diff (71.9 KB ) - added by mir@… 17 years ago.
core changes ported for the unicode branch (for trac, 3rd attempt)
autoescape-1.diff (73.0 KB ) - added by mir@… 17 years ago.
updated patch for svn release 5722
rev5722-01-core-changes.diff (71.2 KB ) - added by mir@… 17 years ago.
updated patch for svn release 5722 (2nd attempt ...)
rev5722-01-core-changes.2.diff (69.7 KB ) - added by mir@… 17 years ago.
updated patch for svn release 5722 (3rd attempt)
rev5722-01-core-changes.3.diff (73.0 KB ) - added by mir@… 17 years ago.
updated patch for svn release 5722 (4th attempt)
rev5722-02-misc-contrib-changes.diff (11.9 KB ) - added by mir@… 17 years ago.
updated patch for svn release 5722
rev5774-01-core-changes.diff (73.1 KB ) - added by mir@… 17 years ago.
updated patch
rev5774-02-misc-contrib-changes.diff (11.9 KB ) - added by mir@… 17 years ago.
updated patch
variable-node.diff (991 bytes ) - added by mir@… 17 years ago.
see comment 17

Download all attachments as: .zip

Change History (41)

by Malcolm Tredinnick, 18 years ago

Attachment: autoescape.diff added

Core changes and test suite

comment:1 by Malcolm Tredinnick, 18 years ago

The mailing list thread for discussing this patch is here. Please use that instead of this ticket for the time being so that the conversation does not happen in two places at once.

comment:2 by Chris Beaven, 18 years ago

Good job on the patch, Malcom :)

A couple of points:
If a markup filter fails due to an import error, I don't think it should be marked as safe.
From a skim read, I'm missing the purpose of having an .is_safe property on filters - can't you just check the outputted string to see if it's SafeData?

comment:3 by Malcolm Tredinnick, 18 years ago

Owner: changed from Adrian Holovaty to Malcolm Tredinnick

by Malcolm Tredinnick, 18 years ago

Attachment: 01-core-changes.diff added

Update core changes (supercedes previous patch)

by Malcolm Tredinnick, 18 years ago

Changes to most of the contrib/ applications (excludes admin)

by Malcolm Tredinnick, 18 years ago

Attachment: 03-admin-changes.diff added

Changes to contrib/admin/

comment:4 by Malcolm Tredinnick, 18 years ago

Owner: changed from Malcolm Tredinnick to Adrian Holovaty

Adrian, this one's back in your court now if you want to give it a review. I think it's complete.

The patches are all independent of each other (each file is only changed by one patch), although 01-core-changes.diff should be applied first. It incorporates all our discussions from OSCON:

  1. off by default
  2. inherits through template extension (so you can mark the "content" in the base template with "autoescape" and be done with it).
  3. adds both "autoescape" and "noautoescape" as per Jacob's suggestion (so that the end tags tell you which tag you are closing explicitly).

The admin change is probably the most destabilising, since mistakes there will be noticed by everybody. It's also probably the hardest one to test in all its variations. I've used it a bit and tried to find problems, but ultimately we are just going to have to be responsive to feedback and fix things as they are noticed. Very few backwards-incompatible changes in the end. The "escape" filter only applying once is the only real one (and we have the new "force_escape" filter if multi-escaping is needed).

comment:5 by Malcolm Tredinnick, 18 years ago

Hmm .. Trac's diff colorizer doesn't like new files being added in the diff. A couple of files appears as being patches to /dev/null. They are the addition of django/utils/safestring.py and tests/othertests/autoescape.py respectively, both in 01-core-changes.diff

The raw diff file has the correct information and can be applied with patch -p1 -i ....

comment:6 by Chris Beaven, 17 years ago

Lest it fly under the Adrian radar, I just want to add that my alternative solution is nearly entirely self-contained, so would require no changes at all to the rest of django if it were to be implemented.

I've been using it for a project (including inside of the existing Admin site) with no hazardous side effects.

comment:7 by Adrian Holovaty, 17 years ago

#2984 was a duplicate.

comment:8 by mir@…, 17 years ago

Patch needs improvement: set
Triage Stage: UnreviewedDesign decision needed

Malcolm stated that this is still work in progress.

by Michael Radziej <mir@…>, 17 years ago

Attachment: 01-core-changes.2.diff added

updated patch

by Michael Radziej <mir@…>, 17 years ago

updated patch

by Michael Radziej <mir@…>, 17 years ago

Attachment: 03-admin-changes.2.diff added

updated patch

comment:9 by Michael Radziej <mir@…>, 17 years ago

I've brought the patches up to date. Bigger changes were:

  • Adapting the tests, this needed some changes in the templates tests. The autoescape tests subclass the templates tests, and this needed a few hooks (but should be ovious)
  • SafeString needs a str() method, or else the one from the normal python string is used, and then it's not SafeData anymore.

It still needs patches for newforms, but this is a fast moving target if I've ever seen one ...

by smurf@…, 17 years ago

Attachment: auto.patch added

don't mark unprocessed markup as safe; fix a string initializer

by Michael Radziej <mir@…>, 17 years ago

Attachment: 01-core-changes.3.diff added

updated patch, includes part of smurf's patch, for rev. 4659

by Michael Radziej <mir@…>, 17 years ago

updated patch, includes part of smurf's patch, for rev. 4659

comment:10 by Michael Radziej <mir@…>, 17 years ago

I've brought the first two patches up to date for revision 3659. They contain smurf's patch, and 01-core also adds empty init.py and models.py files who seemed to have slipped somehow.

It doesn't make sense for me to maintain 03-admin since I don't use the admin anywhere.

comment:11 by Michael Radziej <mir@…>, 17 years ago

patch 01-core-changes: the template tag filter needs to be able to deal with functions decorated by stringfilter (since you cannot use escape or safe with filter).

by Michael Radziej <mir@…>, 17 years ago

Attachment: 01-core-changes.4.diff added

updated patch, see comment

comment:12 by Michael Radziej <mir@…>, 17 years ago

Cc: mir@… added

comment:13 by Chris Beaven, 17 years ago

In the current patch, the example in documentation of is_safe isn't actually safe - the value passed could be a "safe" string which contains a unicode HTML attribute '&#303;' so this filter will introduce a dangerous character ('&').

 	707	 2. If your filter is given a "safe" string, is it guaranteed to return a 
 	708	 "safe" string? If so, set the ``is_safe`` attribute on the function to be 
 	709	 ``True``. For example, a filter that replaced all numbers with the number 
 	710	 spelt out in words is going to be safe-string-preserving, since it cannot 
 	711	 introduce any of the five dangerous characters: <, >, ", ' or &. So we can 
 	712	 write:: 
 	713	 
 	714	    @register.filter 
 	715	    def convert_to_words(value): 
 	716	        # ... implementation here ... 
 	717	        return result 
 	718	 
 	719	    convert_to_words.is_safe = True 

comment:14 by mir@…, 17 years ago

A small bug: Tracebacks for unexpected exceptions in template tags ("Template Error") show a part of the template where the bug hit. With autoescape on, the lines in the template display are escaped one time too often.

by mir@…, 17 years ago

Attachment: unicode-autoescape-1.diff added

core changes ported for the unicode branch

by mir@…, 17 years ago

Attachment: unicode-autoescape-1.2.diff added

core changes ported for the unicode branch (trac-readable)

by mir@…, 17 years ago

Attachment: unicode-autoescape-2.diff added

contrib changes ported for the unicode branch

by mir@…, 17 years ago

Attachment: unicode-autoescape-1.3.diff added

core changes ported for the unicode branch (for trac, 3rd attempt)

comment:15 by mir@…, 17 years ago

I have attached two patches ported to the unicode branch; these positively need further review. Somehow trac refuses to display unicode-autoescape-1, I have no idea why. Sorry for spoiling the history ...

by mir@…, 17 years ago

Attachment: autoescape-1.diff added

updated patch for svn release 5722

by mir@…, 17 years ago

updated patch for svn release 5722 (2nd attempt ...)

by mir@…, 17 years ago

updated patch for svn release 5722 (3rd attempt)

by mir@…, 17 years ago

updated patch for svn release 5722 (4th attempt)

by mir@…, 17 years ago

updated patch for svn release 5722

comment:16 by mir@…, 17 years ago

I brought the first two patches up to date to release 5722. I am aware that trac doesn't parse rev5722-01-core-changes.*.diff, but as much as I tried, I was unable to correct it. Please use rev5722-01-core-changes.3.diff and rev5722-02-misc-contrib-changes.diff

by mir@…, 17 years ago

updated patch

by mir@…, 17 years ago

updated patch

comment:17 by mir@…, 17 years ago

VariableNode.render() must call force_unicode earlier, so that it can see whether it needs to escape or not. I'll add a patch for this separately (variable-node.diff)

BTW: The patches do not (yet) deal with newforms.

by mir@…, 17 years ago

Attachment: variable-node.diff added

see comment 17

comment:18 by Malcolm Tredinnick, 16 years ago

Resolution: fixed
Status: newclosed

(In [6671]) Implemented auto-escaping of variable output in templates. Fully controllable by template authors and it's possible to write filters and templates that simulataneously work in both auto-escaped and non-auto-escaped environments if you need to. Fixed #2359

See documentation in templates.txt and templates_python.txt for how everything
works.

Backwards incompatible if you're inserting raw HTML output via template variables.

Based on an original design from Simon Willison and with debugging help from Michael Radziej.

Note: See TracTickets for help on using tickets.
Back to Top