makemessages command skips the nl-nl-x-informal folder

[Ronnie van den Crommenacker]

There is a mismatch on the DjangoTranslation folder and the makemessages command for the value nl-nl-x-informal

DjangoTranslation uses the to_locale method to determine the language folder to read the django.po. to_locale translates nl-nl-x-informal correctly to the nl_NL-x-informal folder.

However makemessages skips the nl_NL-x-informal folder and displays the following message
invalid locale nl_NL-x-informal, did you mean nl_NL_x_informal?

# This makemessages behaviour is introduced in commit
​https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

The check for - in the locale should only be for the first section a.k.a. nl_NL

[Claude Paroz]

Could you tell us more about the need of this complicated -x-informal stuff?

[Ronnie van den Crommenacker]

It's called a private-use subtag extension of the language_tag.
Source: ​https://en.wikipedia.org/wiki/IETF_language_tag
Source: ​https://www.w3.org/International/articles/language-tags/#extension

An optional private-use subtag, composed of the letter x and a hyphen followed by subtags of one to eight characters each, separated by hyphens.

For our use it's because of our language has a more fomal and informal way of communicating. e.g. "Dear <name>" vs "Hi <name>". With the use of the private-use subtag, we can have 2 variants of the same language.

Ticket about a previous fix for private-use subtags: https://code.djangoproject.com/ticket/28546

[Mariusz Felisiak]

Replying to Ronnie van den Crommenacker:

It's called a private-use subtag extension of the language_tag.
Source: ​https://en.wikipedia.org/wiki/IETF_language_tag
Source: ​https://www.w3.org/International/articles/language-tags/#extension

An optional private-use subtag, composed of the letter x and a hyphen followed by subtags of one to eight characters each, separated by hyphens.

makemessages --locale accepts locales in ISO/IEC 15897 format, not IETF language tags. I'm not sure what private subtags are formatted in ISO/IEC 15897 🤔.

[Ronnie van den Crommenacker]

I cannot find examples of valid ISO/IEC 15897 private subtags either.

But i found some, non standard locales:
​https://www.localeplanet.com/icu/sr-Latn-BA/index.html sr_Latn_BA (this case is mentioned in the django's to_locale comment)
​https://www.localeplanet.com/icu/en-US-POSIX/index.html en_US_POSIX

Also in /usr/share/i18n/SUPPORTED i find some other exceptions
ug_CN@latin
sd_IN@devanagari
ca_ES@valencia

So looking at those examples i would suspect the to_locale should return nl_NL_X_INFORMAL or nl_NL_XINFORMAL or nl_Informal_NL

[Carlton Gibson]

Hi Ronnie.

The check for - in the locale should only be for the first section a.k.a. nl_NL

Do you have a particular patch in mind? Do you want to make a PR?

I'm not sure reading whether this is really supported or not, but if a fix is small and doesn't break anything, then it's easier to think positively about. (If it's disruptive, the opposite might apply...)

[Carlton Gibson]

I'm going to mark this as needs info. If someone can point to clearer specs, or provide a PoC patch, happy to review… (but pending that 🤷)

[Ronnie van den Crommenacker]

Created a pull request: ​https://github.com/django/django/pull/15521

[Carlton Gibson]

Hi Ronnie.

Based on the ​W3.org article, I'd say this isn't something we should really support:

Because these subtags are only meaningful within private agreements and cannot be used interoperably across the Web, they should be used with great care, and avoided whenever possible.

So I would say wontfix.

Then I look at #28546, and see that we did add support in to_locale… — That was your ticket too. I'm guessing you may be the only person using this. 🙂

Can I ask, how come it's taken 5 years for you to hit the issue with makemessages?
Would a project level custom makemessages serve?
This would mean we don't need to add the complication for few (just you?) users 🤔

I have to say I'm still not sure. The patch isn't that big, so we could add it for symmetry with #28546.

[Ronnie van den Crommenacker]

Based on the ​W3.org article, I'd say this isn't something we should really support:

Because these subtags are only meaningful within private agreements and cannot be used interoperably across the Web, they should be used with great care, and avoided whenever possible.

So I would say wontfix.

The way i read the comment is "it should be ~used~ with great care", which doesn't not say anything about the ~support~ for the private extension.
It exists in the language-tags for a reason, but for locales it seems not that clear in ISO/IEC 15897 what the restrictions to the format actually are. I have seen some uncommon formats in the wild (see comment https://code.djangoproject.com/ticket/33565?replyto=9#comment:5).

I understand that if something should not be used there is little reason to support it. However, Django has added support for private subtags in the to_locale function already via #28546 and it used to work before Django 3.2.

Then I look at #28546, and see that we did add support in to_locale… — That was your ticket too. I'm guessing you may be the only person using this. 🙂

Yeah, i think i'm the only one. It was (and is) still the perfect solution for the problem we (as a company) are facing. Other solutions would be far more complex.

Can I ask, how come it's taken 5 years for you to hit the issue with makemessages?
Would a project level custom makemessages serve?
This would mean we don't need to add the complication for few (just you?) users 🤔

The makemessages command worked (maybe by accident) perfectly for many years, until we updated to Django 3.2 recently.
This commit included in Django 3.2 added extra checks which, by accident, stopped the support for private subtags
​​https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

As a temporary solution we rename nl_NL_x_informal to nl_NL-x-informal after we run makemessages

I have to say I'm still not sure. The patch isn't that big, so we could add it for symmetry with #28546.

I hope you will reconsider to include the patch. I'll understand Django wouldn't support it officially, document it and test for it in upcoming releases. If it occasionally breaks, ill understand and i am happy to contribute patches when it breaks again.

[Carlton Gibson]

... i am happy to contribute patches when it breaks again.

OK, let's take it for review then :) If no-one objects to the PR, then you can be on the hook to help maintain it. 😄

[Carlton Gibson]

Patch is updated. Thanks Ronnie. Unchecking for another review.

[Carlton Gibson <carlton.gibson@…>]

In c32858a8:

Fixed #33565 -- Improved locale format validation for the makemessages command.