yesno filter broken for translations that do not respect the ascii comma

[Klaas van Schelven]

2 related problems:

1: Some languages do not translate the "yes,no,maybe" string including comma's

$ find -name "*.po" | x grep yes,no,maybe -A1 | grep msgstr | grep -v , | grep -v '""' 
./django/conf/locale/km/LC_MESSAGES/django.po-msgstr "យល់ព្រម មិនយល់ព្រម​ ប្រហែល"
./django/conf/locale/zh_Hant/LC_MESSAGES/django.po-msgstr "是、否、也許"
./django/conf/locale/fa/LC_MESSAGES/django.po-msgstr "بله،خیر،شاید"
./django/conf/locale/ur/LC_MESSAGES/django.po-msgstr "ھاں،نہیں،ھوسکتاہے"
./django/conf/locale/ug/LC_MESSAGES/django.po-msgstr "ھەئە، ياق، بەلكىىم"
./django/conf/locale/ckb/LC_MESSAGES/django.po-msgstr "بەڵێ،نەخێر،لەوانەیە"
./django/conf/locale/zh_Hans/LC_MESSAGES/django.po-msgstr "是、否、也许"

2. the filter returns the value when the arg is incorrect: ​https://github.com/django/django/blob/main/django/template/defaultfilters.py#L881

IMHO this is wrong, because it breaks the Maybe(bool) => str type of the function. For the broken arg, it should either:
[a] override the arg to the default "yes,no,maybe"
[b] raise an exception

these 2 things combined make it so that when any of the broken languages is activated, the filter is broken. Consider e.g. yesno(True).capitalize() which will now raise an exception in Chinese.

[Klaas van Schelven]

Looked at this a bit more.

The “missing comma” issue seems to have two causes across translations:

• Some locales use spaces instead of commas (e.g. Khmer).
• Others use locale-specific commas (e.g. Chinese 、, Persian/Urdu/Uyghur/Kurdish ،).

So:

1. the updates to the locales should in most cases be replacing a local comma into the latin/ascii one.
2. To avoid this recurring, the translator note could be strengthened to: “Use ASCII commas (,) with no spaces for machine parsing. Do not substitute local punctuation.” (or similar)

[Natalia Bidart]

Thank you Klaas van Schelven! Your report makes sense but ideally we would need clear instructions in how to reproduce. Would you be able to provide a failing test case? That would certainly help us with the triage.

[Klaas van Schelven]

I've provided a failing test-case which I've then fixed, ​https://github.com/django/django/pull/19791

As per the PR: that's part of the work; I will leave fixing of the translations to the people who are on that platform (I am not and I don't want to go out of my way to create an account there).

[Natalia Bidart]

Thank you! The PR certainly helps understanding your goal.

I agree we should return a safe string as default and we should also fix the translations. I'll do the later and review your PR soon.

[Natalia Bidart]

I have updated all translations, the next time we fetch updated translations into Django, these will be available.

[Natalia Bidart]

Claude, would you have ideas in how we could prevent this issue with the (lack of) ascii commas in the future?

[Claude Paroz]

I guess I would:

• make the translator comment clearer about using ascii commas
• program a check in the manage_translations.py script