[unicode] Unicode-aware translation

[Ivan Sagalaev <Maniac@…>]

I've made two flavors of a patch. First one only fixes template tags for translations (and is very small). Another includes the first one plus replaces [n]gettext* -> u[n]gettext in db fields, newforms fields. Things to think of:

• Malcolm, despite your observations _(u'something in russian') doesn't break anything including make-messages.py. But I didn't change arguments to _() because they should be in English anyway
• newforms' fields.py uses gettext(u'...') all over the place. I think they should be converted to '...' for consistency but this didn't make it into the patch. If there are no objections I'll convert them.
• I didn't touch core.validators at all because they are intended for oldforms that aren't unicode aware. However they are used in db.models.fields (EMailField for example). After converting things to ugettext this leads to some validation messages being in unicode and some in byte strings (see tests/modeltests/validation/models.py lines 45 -- unicode and 147 -- byte strings). I believe the best way is to convert core.validators to unicode and forget oldforms. Thoughts?

[Ivan Sagalaev <Maniac@…>]

Patch for template tags

[Ivan Sagalaev <Maniac@…>]

Cumulative patch

[Malcolm Tredinnick]

Ivan,

• Not sure why you aren't seeing errors in make-messages.py. I get them all the time with both gettext 0.14.5 and gettext 0.16.1. Most odd. It's documented as not valid for gettext msg ids anyway.
• There's no problem with using u".." strings inside the gettext() calls providing they use ASCII characters. xgettext only looks for the quotes anyway. I haven't been bothering to change them in localflavor/ contributions.
• Old forms is deprecated, not absent. We need to fix oldforms, too, for backwards compatibility. I don't know if oldforms will still be present in 1.0 or not (I would guess, yes, but import forms will import newforms by default). We can't break the tree once unicode is merged with trunk and there's a lot of code still using oldforms. I'll fix that up, though, it's not too hard, I suspect.

Patch looks good from just reading it through once. I'll merge it tonight. Thanks.

(Oh, and stop putting [patch] in the title. There's a checkbox for that. Using "[patch]" makes it look like you're still living in 2006.)

[Ivan Sagalaev <Maniac@…>]

• Not sure why you aren't seeing errors in make-messages.py. I get them all the time with both gettext 0.14.5 and gettext 0.16.1. Most odd. It's documented as not valid for gettext msg ids anyway.

Yes, I just noted this as a strange fact, not for using it :-)

• There's no problem with using u".." strings inside the gettext() calls providing they use ASCII characters. xgettext only looks for the quotes anyway. I haven't been bothering to change them in localflavor/ contributions.

I thought that this may be misleading for people reading source and thinking "if it can accept unicode I could use my native language as a base for translation".

• Old forms is deprecated, not absent. We need to fix oldforms, too, for backwards compatibility. I don't know if oldforms will still be present in 1.0 or not (I would guess, yes, but import forms will import newforms by default). We can't break the tree once unicode is merged with trunk and there's a lot of code still using oldforms. I'll fix that up, though, it's not too hard, I suspect.

I actually have core.validators converted locally, I'll just update the patch in a few hours.

(Oh, and stop putting [patch] in the title. There's a checkbox for that. Using "[patch]" makes it look like you're still living in 2006.)

Ouch :-). Perhaps I was trying too hard to get used to it and now it strikes back :-).

[Ivan Sagalaev <Maniac@…>]

Cumulative patch including oldforms and core.validators

[Malcolm Tredinnick]

(In [5082]) unicode: Fixed #4152 -- Converted a lot of internal [n]gettext() calls into
u[n]gettext() calls. Thanks, Ivan Sagalaev.