Opened 12 years ago

Closed 12 years ago

#3977 closed (fixed)

[unicode] Convert default filters

Reported by: Ivan Sagalaev <Maniac@…> Owned by: Malcolm Tredinnick
Component: Template system Version: master
Severity: Keywords: unicode
Cc: Maniac@…, gabor@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no


Default filters think they should deal with strings.

I'm in the process of making a patch.

Attachments (1)

3977.diff (26.5 KB) - added by Ivan Sagalaev <Maniac@…> 12 years ago.

Download all attachments as: .zip

Change History (9)

comment:1 Changed 12 years ago by Malcolm Tredinnick

Cc: Malcolm Tredinnick removed
Owner: changed from Adrian Holovaty to Malcolm Tredinnick

They should still be able to deal with bytestrings. They also need to be able to deal with unicode strings.

The idea isn't to only handle unicode, it's to be able to also handle unicode. Otherwise the pain for developers is too great because often bytestrings will work, as they come naturally encoded in a format we can use (e.g. ASCII or UTF-8).

comment:2 Changed 12 years ago by anonymous

Cc: gabor@… added

if the filters have to be able to work with bytestrings too, then what will they output? only unicode?
or it depends on their input?

and if they have to handle bytestrings too, how should they handle them? just calling smart_unicode on
the input should be enough?

also, why do you think the pain would be too great, if the templates would require all their input to be in unicode?
for me it seems to be a cleaner, more explicit approach...

comment:3 Changed 12 years ago by Ivan Sagalaev <Maniac@…>

In my current code I have all the "stringfilters" pass their input through smart_unicode (that Malcolm has added in django.utils.encoding). This ensures that filters accept both unicode and byte-strings in utf-8. A byte-string fallback is useful mostly for model objects that don't define unicode and have only str representation.

However I agree (and already implemented) that all "stringfilters" output only unicode objects.

Any objections?

comment:4 Changed 12 years ago by Malcolm Tredinnick

Sounds good, Ivan. Outputting unicode is fine, because it's going straight back into the template.

We have to be able to accept bytestring input because it could be coming from a method on the model or anywhere. Calling smart_unicode once on input is the approach I would take.

Drop in the patch when you're happy and I'll merge it. I'm staying away from that piece of code while you're working on it.

Changed 12 years ago by Ivan Sagalaev <Maniac@…>

Attachment: 3977.diff added


comment:5 Changed 12 years ago by Ivan Sagalaev <Maniac@…>

Summary: [unicode] Convert default filters[patch] [unicode] Convert default filters

Attached a patch. In couple of places there are comments starting with # isagalaev: about fixing underlying utilities instead of filters themselves (e.g. utils.text.truncate_html_words returns plain str even if given unicode on input). Questions are:

  • is it really better or we don't want to move text utils to unicode?
  • if not, should I remove comments or let them be there for future work?

comment:6 Changed 12 years ago by Simon G. <dev@…>

Has patch: set
Keywords: unicode added
Patch needs improvement: set
Summary: [patch] [unicode] Convert default filters[unicode] Convert default filters
Triage Stage: UnreviewedReady for checkin

comment:7 Changed 12 years ago by Malcolm Tredinnick

Ivan, this patch is all good. I've been merging it in with the code today. Taking some time, since I think you're right and utils.text and friends should be unicode aware as well and that is leading to cascading problems (have to fix i18n support first, etc).

So this will go in as part of a larger commit shortly. No more work required from you on this. Thanks a lot for doing it.

comment:8 Changed 12 years ago by Malcolm Tredinnick

Resolution: fixed
Status: newclosed

(In [5056]) Changed default template filters to always return unicode (and to handle unicode input better). Also changed all django.utils.* functions that are used (or likely to be used) by filters to return unicode strings.

Most of the filter porting was done by Ivan Sagalaev. Fixed #3977.

Note: See TracTickets for help on using tickets.
Back to Top