#11240 closed Bug (fixed)
Compilemessages fails if a % character is at certain places in the .po file
Reported by: | Till Backhaus | Owned by: | Marc Garcia |
---|---|---|---|
Component: | Internationalization | Version: | 1.3 |
Severity: | Normal | Keywords: | |
Cc: | kikko, harm.verhagen+django@…, claude@… | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | yes |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
Reproduce:
Create a template:
{% load i18n %} {% trans "findme 10% " %}
run ./manage.py makemessages -a
find the string in the .po file and translate it like this:
#: templates/test.html:2 #, python-format msgid "findme 10% of their" msgstr "findemich 10% an "
run ./manage.py compilemessages
get this error message:
/project/locale/de/LC_MESSAGES/django.po:925: 'msgstr' is not a valid Python format string, unlike 'msgid'. Reason: In the directive number 1, the character 'a' is not a valid conversion specifier. msgfmt: found 1 fatal errors
Attachments (2)
Change History (34)
comment:1 by , 15 years ago
Description: | modified (diff) |
---|
comment:2 by , 15 years ago
Yes, for the translatable literals extraction process templates are (internally) converted to python code and then fed as such to the gettext tools.
This means that Python string formatting specifiers rules should be followed when using the i18n template tags with string literals containing extrapolated variables or special meaning chars like %
. In this case it means you literal would need to be {% trans "findme 10%% " %}
as per http://www.gnu.org/software/gettext/manual/gettext.html#python_002dformat and http://www.python.org/doc/2.2.1/lib/typesseq-strings.html (linked from the first).
I don't know if this means we should add a note to the effect to the docs.
comment:3 by , 15 years ago
Owner: | changed from | to
---|
comment:4 by , 15 years ago
Component: | Internationalization → Documentation |
---|---|
Triage Stage: | Unreviewed → Accepted |
comment:5 by , 14 years ago
For me that outputs "findme 10%%" and not "findme 10%" so that doesn't really seem to be a solution.
comment:6 by , 14 years ago
Resolution: | → worksforme |
---|---|
Status: | new → closed |
Note the reporter claims he extracted the translatable literal from a template but it is prefixed by a #, python-format
flag in the PO file, and that's not consistent.
I've tested both kinds (with and without a python-format
flag) of literals and translations always similar to the ones reported and didn't get any compilemessages error. MO file got generated without error.
Closing this old ticket as worksforme instead of fixed because I don't see any change committed to the compilemessages
management command related to this.
follow-up: 11 comment:7 by , 14 years ago
Component: | Documentation → Translations |
---|---|
Easy pickings: | unset |
Resolution: | worksforme |
Severity: | → Normal |
Status: | closed → reopened |
Type: | → Uncategorized |
Version: | 1.0 → 1.3 |
@ramiro:
Note the reporter claims he extracted the translatable literal from a template but it is prefixed by a #, python-format flag in the PO file, and that's not consistent.
That's very consistent: It is makemessages that generated this flag, because it saw a '%' in the string!
because I don't see any change committed to the compilemessages management command related to this.
Sorry, but that's not enough. You're right that the error message will go away. Nonetheless, two percentage characters are outputted when using {% trans "findme 10%%" %}: "findme 10%%"
Reopening. I have reproduced it on Django 1.3 and 1.2. As of now, the fix is to use blocktrans instead, in which case you write:
{% blocktrans %}findme 10%{% endblocktrans %}
...producing correct .po gramma:
#, python-format msgid "findme 10%%" msgstr "findme 10%%"
...and it will wrongly output an untranslated "findme 10%" -- it doesn't translate!
comment:9 by , 13 years ago
Type: | Uncategorized → Bug |
---|
comment:10 by , 13 years ago
Cc: | added |
---|
comment:11 by , 13 years ago
Resolution: | → needsinfo |
---|---|
Status: | reopened → closed |
Replying to ramiro:
Yes, for the translatable literals extraction process templates are (internally) converted to python code and then fed as such to the gettext tools.
This means that Python string formatting specifiers rules should be followed when using the i18n template tags with string literals containing extrapolated variables or special meaning chars like
%
. In this case it means you literal would need to be{% trans "findme 10%% " %}
I've verified this is behaving correctly with trunk as of now and with 1.3. Ignore that advice I gave back then. I suspect r14459 fixed this in django so it works transparently for the app developer without the need to use %%
with the trans
i18n template tag.
To test things I created a template like this:
{% load i18n %} {% trans "a literal with a percent symbol at the end %" %}</br> {% trans "a literal with a percent symbol at the end 10%" %}</br> {% trans "a literal with a percent % symbol in the middle" %}</br> {% trans "a literal with a percent 20% symbol in the middle" %}</br>
makemessages -l de
created a .po
file like this from it (note I've already added dummy translations):
#: t11240/templates/a.html:2 msgid "a literal with a percent symbol at the end %" msgstr "translation to German of a literal with a percent symbol at the end %" #: t11240/templates/a.html:3 msgid "a literal with a percent symbol at the end 10%" msgstr "" "translation to German of a literal with a percent symbol at the end 10%" #: t11240/templates/a.html:4 #, python-format msgid "a literal with a percent % symbol in the middle" msgstr "" "translation to German of a literal with a percent % symbol in the middle" #: t11240/templates/a.html:5 #, python-format msgid "a literal with a percent 20% symbol in the middle" msgstr "" "translation to German a literal with a percent 20% symbol in the middle"
and compilemessages
creates a corresponding .mo
file without problems.
Setting LANGUAGE="de"
and using the template in a view correctly shows:
translation to German of a literal with a percent symbol at the end % translation to German of a literal with a percent symbol at the end 10% translation to German of a literal with a percent % symbol in the middle translation to German a literal with a percent 20% symbol in the middle
I'm closing this ticket and setting the reason needsinfo. If you reopen it please provide exact details of what you are seeing.
comment:12 by , 13 years ago
Component: | Translations → Internationalization |
---|
comment:13 by , 13 years ago
Resolution: | needsinfo |
---|---|
Status: | closed → reopened |
UI/UX: | unset |
Reopened.
I see this issue too with django 1.3
template
{% trans '% of test' %}
po file
msgid "% of test" msgstr "% van testtrans"
python manage.py compilemessages
gives the following error
locale/nl/LC_MESSAGES/django.po:950: 'msgstr' is not a valid Python format string, unlike 'msgid'. Reason: In the directive number 1, the character 'v' is not a valid conversion specifier. msgfmt: found 1 fatal error
Escaping with a double %%
in the template doesn'th help. (that renders as a double percentage)
comment:14 by , 13 years ago
Cc: | added |
---|
comment:15 by , 13 years ago
I think I understand now what this ticket has always been about.
The issue is with the '% o'
fragment of the examples in the msgid's provided by both users that experienced problems.
'% o'
is a valid interpolation specification, it mean an unsigned octal preceded by a space for positive values (or the '-'
sign for negative ones) this is what the space conversion flag means.
I suspect this isn't the only case when an unintended formatting specifier can sneak in translatable literals involving percent characters.
comment:16 by , 13 years ago
Has patch: | set |
---|
The patch attached implements a fix for this in the 'makemessages' step by implementing escaping of '%' symbols in literals passed to the {% trans %}
template tag (it replaces them with '%%'
).
This means that starting with this change simple Python string interpolation isn't supported in literals passed to 'trans'
anymore.
This also means that the 'msgid'
's extracted from such literals will have now '%%'
and that translators should also use the same sequence in the respective 'msgstr'
's. With this:
- Unfortunately GNU gettext's xgettext still marks the msgid/msgstr entry with the
#, python-format
flag. Even when it contains no Python string formatting specification (e.g. "A string with two %%" still gets marked so). There is no way we can avoid addition of the flag or to remove it afterward because the execution of xgettext is an opaque step. - Even when the entry is marked with the
#, python-format
flag GNU gettext's msgfmt command additional checks triggered by such mark doesn't reject it with the error message reported in this ticket.
by , 13 years ago
Attachment: | 11240-1.diff added |
---|
comment:18 by , 13 years ago
Triage Stage: | Accepted → Ready for checkin |
---|
comment:19 by , 13 years ago
Cc: | added |
---|
comment:21 by , 12 years ago
I still see this with 1.4.3? All msgstr's with % fail as original bug explains.
comment:23 by , 12 years ago
Resolution: | fixed |
---|---|
Status: | closed → new |
I have issues with trans tags containing % Symbols using django 1.5.1
The trans tag doesn't handle a % the same way makemessages does. A {% trans "value in %"}
still looks for a msgid "value in %"
but makemessages created a msgid "value in %%"
for it.
The trans tag should implement the same string conversion as makemessage does.
The workaround to already escape the % in the trans tag - {% trans "value in %%"}
- returns the right translation.
But as the translation will most likely have a "%%" in it the rendered conent will have a "%%" too.
A workaround could be to let the TranslateNode return value % ()
.
Currently the only way to avoid issues with % Symbols in translations is using blocktrans tags.
It might be worth monitoring http://savannah.gnu.org/bugs/?func=detailitem&item_id=30854 for changes on gettext handling python format strings.
by , 12 years ago
Attachment: | 11240-test.diff added |
---|
Test showing different behaviour between trans and blocktrans
comment:24 by , 12 years ago
Unfortunately yes, I also think there is still an issue with trans
and percents. Just attached the test case as it should be (and which currently fails).
comment:25 by , 12 years ago
You could just use a custom wrapper for pgettext (for example)
wrapper.py:
from django.utils.translation import pgettext as pgt
def pgettext(context, msg):
return pgt(context, msg).replace("%%", "%")
comment:26 by , 12 years ago
Another workaround is to use alternative syntax for translation:
{{ _('Percent sign here % is not escaped.') }}
I guess that gettext might be looking for python syntaxed string-formatting in template files and is mislead because text string is followed directly by percent sign %}.
comment:27 by , 11 years ago
Patch needs improvement: | set |
---|---|
Triage Stage: | Ready for checkin → Accepted |
comment:28 by , 11 years ago
It would have been much better if you had opened a new ticket instead of reopening this one.
I suspect the issue has to do with the fact that the xgettext tool unconditionally adds a "#, python-format"
flag (see comment:16) but we (by design) don't support Python string format specifiers on 'trans' tag literals.
follow-up: 30 comment:29 by , 11 years ago
Trying to replicate the bug, I've found that #, python-format
compiles just fine with invalid string format specifiers.
#, python-format msgid "My project has 20% success" msgstr "No such thing as 20% success"
$ xgettext -V xgettext (GNU gettext-tools) 0.18.2
Why would this be the case? Is this expected behaviour? If so, replacing %
by %%
makes no sense and that changeset should be reverted.
comment:30 by , 11 years ago
Replying to bouke:
Trying to replicate the bug, I've found that
#, python-format
compiles just fine with invalid string format specifiers.
#, python-format msgid "My project has 20% success" msgstr "No such thing as 20% success"
These aren't invalid Python format specifiers. "% s" is a valid one. Can you re-test with a %-prefixed sequence that is actually invalid?
comment:31 by , 11 years ago
I'm re-closing this ticket because the original issue was fixed two years ago.
If you intend to follow-up please open a newer one with a clean slate. Make sure you take in account comments comment:28 and comment:30 in you analysis.
comment:32 by , 11 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
(formatted description)