[unicode] Convert default filters

They should still be able to deal with bytestrings. They also need to be able to deal with unicode strings.

The idea isn't to only handle unicode, it's to be able to also handle unicode. Otherwise the pain for developers is too great because often bytestrings will work, as they come naturally encoded in a format we can use (e.g. ASCII or UTF-8).

if the filters have to be able to work with bytestrings too, then what will they output? only unicode? or it depends on their input?

and if they have to handle bytestrings too, how should they handle them? just calling smart_unicode on the input should be enough?

also, why do you think the pain would be too great, if the templates would require all their input to be in unicode? for me it seems to be a cleaner, more explicit approach...

In my current code I have all the "stringfilters" pass their input through smart_unicode (that Malcolm has added in django.utils.encoding). This ensures that filters accept both unicode and byte-strings in utf-8. A byte-string fallback is useful mostly for model objects that don't define unicode and have only str representation.

However I agree (and already implemented) that all "stringfilters" output only unicode objects.

Any objections?

Sounds good, Ivan. Outputting unicode is fine, because it's going straight back into the template.

We have to be able to accept bytestring input because it could be coming from a method on the model or anywhere. Calling smart_unicode once on input is the approach I would take.

Drop in the patch when you're happy and I'll merge it. I'm staying away from that piece of code while you're working on it.

Patch

Attached a patch. In couple of places there are comments starting with # isagalaev: about fixing underlying utilities instead of filters themselves (e.g. utils.text.truncate_html_words returns plain str even if given unicode on input). Questions are:

• is it really better or we don't want to move text utils to unicode?
• if not, should I remove comments or let them be there for future work?

Ivan, this patch is all good. I've been merging it in with the code today. Taking some time, since I think you're right and utils.text and friends should be unicode aware as well and that is leading to cascading problems (have to fix i18n support first, etc).

So this will go in as part of a larger commit shortly. No more work required from you on this. Thanks a lot for doing it.

(In [5056]) Changed default template filters to always return unicode (and to handle unicode input better). Also changed all django.utils.* functions that are used (or likely to be used) by filters to return unicode strings.

Most of the filter porting was done by Ivan Sagalaev. Fixed #3977.