Opened 9 years ago

Closed 8 years ago

#3977 closed (fixed)

[unicode] Convert default filters

Reported by: Ivan Sagalaev <Maniac@…> Owned by: mtredinnick
Component: Template system Version: master
Severity: Keywords: unicode
Cc: Maniac@…, gabor@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:


Default filters think they should deal with strings.

I'm in the process of making a patch.

Attachments (1)

3977.diff (26.5 KB) - added by Ivan Sagalaev <Maniac@…> 9 years ago.

Download all attachments as: .zip

Change History (9)

comment:1 Changed 9 years ago by mtredinnick

  • Cc mtredinnick removed
  • Needs documentation unset
  • Needs tests unset
  • Owner changed from adrian to mtredinnick
  • Patch needs improvement unset

They should still be able to deal with bytestrings. They also need to be able to deal with unicode strings.

The idea isn't to only handle unicode, it's to be able to also handle unicode. Otherwise the pain for developers is too great because often bytestrings will work, as they come naturally encoded in a format we can use (e.g. ASCII or UTF-8).

comment:2 Changed 9 years ago by anonymous

  • Cc gabor@… added

if the filters have to be able to work with bytestrings too, then what will they output? only unicode?
or it depends on their input?

and if they have to handle bytestrings too, how should they handle them? just calling smart_unicode on
the input should be enough?

also, why do you think the pain would be too great, if the templates would require all their input to be in unicode?
for me it seems to be a cleaner, more explicit approach...

comment:3 Changed 9 years ago by Ivan Sagalaev <Maniac@…>

In my current code I have all the "stringfilters" pass their input through smart_unicode (that Malcolm has added in django.utils.encoding). This ensures that filters accept both unicode and byte-strings in utf-8. A byte-string fallback is useful mostly for model objects that don't define unicode and have only str representation.

However I agree (and already implemented) that all "stringfilters" output only unicode objects.

Any objections?

comment:4 Changed 9 years ago by mtredinnick

Sounds good, Ivan. Outputting unicode is fine, because it's going straight back into the template.

We have to be able to accept bytestring input because it could be coming from a method on the model or anywhere. Calling smart_unicode once on input is the approach I would take.

Drop in the patch when you're happy and I'll merge it. I'm staying away from that piece of code while you're working on it.

Changed 9 years ago by Ivan Sagalaev <Maniac@…>


comment:5 Changed 9 years ago by Ivan Sagalaev <Maniac@…>

  • Summary changed from [unicode] Convert default filters to [patch] [unicode] Convert default filters

Attached a patch. In couple of places there are comments starting with # isagalaev: about fixing underlying utilities instead of filters themselves (e.g. utils.text.truncate_html_words returns plain str even if given unicode on input). Questions are:

  • is it really better or we don't want to move text utils to unicode?
  • if not, should I remove comments or let them be there for future work?

comment:6 Changed 9 years ago by Simon G. <dev@…>

  • Has patch set
  • Keywords unicode added
  • Patch needs improvement set
  • Summary changed from [patch] [unicode] Convert default filters to [unicode] Convert default filters
  • Triage Stage changed from Unreviewed to Ready for checkin

comment:7 Changed 8 years ago by mtredinnick

Ivan, this patch is all good. I've been merging it in with the code today. Taking some time, since I think you're right and utils.text and friends should be unicode aware as well and that is leading to cascading problems (have to fix i18n support first, etc).

So this will go in as part of a larger commit shortly. No more work required from you on this. Thanks a lot for doing it.

comment:8 Changed 8 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [5056]) Changed default template filters to always return unicode (and to handle unicode input better). Also changed all django.utils.* functions that are used (or likely to be used) by filters to return unicode strings.

Most of the filter porting was done by Ivan Sagalaev. Fixed #3977.

Note: See TracTickets for help on using tickets.
Back to Top