Opened 7 months ago

Closed 4 months ago

#33565 closed Bug (fixed)

makemessages command skips the nl-nl-x-informal folder

Reported by: Ronnie van den Crommenacker Owned by: Ronnie van den Crommenacker
Component: Internationalization Version: 3.2
Severity: Normal Keywords:
Cc: Claude Paroz Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

There is a mismatch on the DjangoTranslation folder and the makemessages command for the value nl-nl-x-informal

DjangoTranslation uses the to_locale method to determine the language folder to read the django.po. to_locale translates nl-nl-x-informal correctly to the nl_NL-x-informal folder.

However makemessages skips the nl_NL-x-informal folder and displays the following message
invalid locale nl_NL-x-informal, did you mean nl_NL_x_informal?

# This makemessages behaviour is introduced in commit
https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

The check for - in the locale should only be for the first section a.k.a. nl_NL

Change History (16)

comment:1 Changed 7 months ago by Mariusz Felisiak

Cc: Claude Paroz added

comment:2 Changed 7 months ago by Claude Paroz

Could you tell us more about the need of this complicated -x-informal stuff?

comment:3 Changed 7 months ago by Ronnie van den Crommenacker

It's called a private-use subtag extension of the language_tag.
Source: https://en.wikipedia.org/wiki/IETF_language_tag
Source: https://www.w3.org/International/articles/language-tags/#extension

An optional private-use subtag, composed of the letter x and a hyphen followed by subtags of one to eight characters each, separated by hyphens.

For our use it's because of our language has a more fomal and informal way of communicating. e.g. "Dear <name>" vs "Hi <name>". With the use of the private-use subtag, we can have 2 variants of the same language.

Ticket about a previous fix for private-use subtags: https://code.djangoproject.com/ticket/28546

Last edited 7 months ago by Ronnie van den Crommenacker (previous) (diff)

comment:4 in reply to:  3 Changed 7 months ago by Mariusz Felisiak

Replying to Ronnie van den Crommenacker:

It's called a private-use subtag extension of the language_tag.
Source: https://en.wikipedia.org/wiki/IETF_language_tag
Source: https://www.w3.org/International/articles/language-tags/#extension

An optional private-use subtag, composed of the letter x and a hyphen followed by subtags of one to eight characters each, separated by hyphens.

makemessages --locale accepts locales in ISO/IEC 15897 format, not IETF language tags. I'm not sure what private subtags are formatted in ISO/IEC 15897 🤔.

comment:5 Changed 7 months ago by Ronnie van den Crommenacker

I cannot find examples of valid ISO/IEC 15897 private subtags either.

But i found some, non standard locales:
https://www.localeplanet.com/icu/sr-Latn-BA/index.html sr_Latn_BA (this case is mentioned in the django's to_locale comment)
https://www.localeplanet.com/icu/en-US-POSIX/index.html en_US_POSIX

Also in /usr/share/i18n/SUPPORTED i find some other exceptions
ug_CN@latin
sd_IN@devanagari
ca_ES@valencia

So looking at those examples i would suspect the to_locale should return nl_NL_X_INFORMAL or nl_NL_XINFORMAL or nl_Informal_NL or nl_NL@informal

Last edited 7 months ago by Ronnie van den Crommenacker (previous) (diff)

comment:6 Changed 7 months ago by Carlton Gibson

Hi Ronnie.

The check for - in the locale should only be for the first section a.k.a. nl_NL

Do you have a particular patch in mind? Do you want to make a PR?

I'm not sure reading whether this is really supported or not, but if a fix is small and doesn't break anything, then it's easier to think positively about. (If it's disruptive, the opposite might apply...)

comment:7 Changed 7 months ago by Carlton Gibson

Resolution: needsinfo
Status: newclosed

I'm going to mark this as needs info. If someone can point to clearer specs, or provide a PoC patch, happy to review… (but pending that 🤷)

comment:8 Changed 7 months ago by Ronnie van den Crommenacker

Has patch: set
Resolution: needsinfo
Status: closednew

comment:9 Changed 6 months ago by Carlton Gibson

Hi Ronnie.

Based on the W3.org article, I'd say this isn't something we should really support:

Because these subtags are only meaningful within private agreements and cannot be used interoperably across the Web, they should be used with great care, and avoided whenever possible.

So I would say wontfix.

Then I look at #28546, and see that we did add support in to_locale… — That was your ticket too. I'm guessing you may be the only person using this. 🙂

Can I ask, how come it's taken 5 years for you to hit the issue with makemessages?
Would a project level custom makemessages serve?
This would mean we don't need to add the complication for few (just you?) users 🤔

I have to say I'm still not sure. The patch isn't that big, so we could add it for symmetry with #28546.


comment:10 in reply to:  9 Changed 6 months ago by Ronnie van den Crommenacker

Based on the W3.org article, I'd say this isn't something we should really support:

Because these subtags are only meaningful within private agreements and cannot be used interoperably across the Web, they should be used with great care, and avoided whenever possible.

So I would say wontfix.

The way i read the comment is "it should be ~used~ with great care", which doesn't not say anything about the ~support~ for the private extension.
It exists in the language-tags for a reason, but for locales it seems not that clear in ISO/IEC 15897 what the restrictions to the format actually are. I have seen some uncommon formats in the wild (see comment https://code.djangoproject.com/ticket/33565?replyto=9#comment:5).

I understand that if something should not be used there is little reason to support it. However, Django has added support for private subtags in the to_locale function already via #28546 and it used to work before Django 3.2.

Then I look at #28546, and see that we did add support in to_locale… — That was your ticket too. I'm guessing you may be the only person using this. 🙂

Yeah, i think i'm the only one. It was (and is) still the perfect solution for the problem we (as a company) are facing. Other solutions would be far more complex.

Can I ask, how come it's taken 5 years for you to hit the issue with makemessages?
Would a project level custom makemessages serve?
This would mean we don't need to add the complication for few (just you?) users 🤔

The makemessages command worked (maybe by accident) perfectly for many years, until we updated to Django 3.2 recently.
This commit included in Django 3.2 added extra checks which, by accident, stopped the support for private subtags
https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

As a temporary solution we rename nl_NL_x_informal to nl_NL-x-informal after we run makemessages

I have to say I'm still not sure. The patch isn't that big, so we could add it for symmetry with #28546.

I hope you will reconsider to include the patch. I'll understand Django wouldn't support it officially, document it and test for it in upcoming releases. If it occasionally breaks, ill understand and i am happy to contribute patches when it breaks again.

comment:11 Changed 6 months ago by Carlton Gibson

Triage Stage: UnreviewedAccepted

... i am happy to contribute patches when it breaks again.

OK, let's take it for review then :) If no-one objects to the PR, then you can be on the hook to help maintain it. 😄

comment:12 Changed 5 months ago by Mariusz Felisiak

Owner: changed from nobody to Ronnie van den Crommenacker
Status: newassigned

comment:13 Changed 4 months ago by Carlton Gibson

Needs tests: set

comment:14 Changed 4 months ago by Carlton Gibson

Needs tests: unset

Patch is updated. Thanks Ronnie. Unchecking for another review.

comment:15 Changed 4 months ago by Carlton Gibson

Triage Stage: AcceptedReady for checkin

comment:16 Changed 4 months ago by Carlton Gibson <carlton.gibson@…>

Resolution: fixed
Status: assignedclosed

In c32858a8:

Fixed #33565 -- Improved locale format validation for the makemessages command.

Note: See TracTickets for help on using tickets.
Back to Top