Opened 2 years ago

Closed 22 months ago

#33565 closed Bug (fixed)

makemessages command skips the nl-nl-x-informal folder

Reported by: Ronnie van den Crommenacker Owned by: Ronnie van den Crommenacker
Component: Internationalization Version: 3.2
Severity: Normal Keywords:
Cc: Claude Paroz Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

There is a mismatch on the DjangoTranslation folder and the makemessages command for the value nl-nl-x-informal

DjangoTranslation uses the to_locale method to determine the language folder to read the django.po. to_locale translates nl-nl-x-informal correctly to the nl_NL-x-informal folder.

However makemessages skips the nl_NL-x-informal folder and displays the following message
invalid locale nl_NL-x-informal, did you mean nl_NL_x_informal?

# This makemessages behaviour is introduced in commit
https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

The check for - in the locale should only be for the first section a.k.a. nl_NL

Change History (16)

comment:1 by Mariusz Felisiak, 2 years ago

Cc: Claude Paroz added

comment:2 by Claude Paroz, 2 years ago

Could you tell us more about the need of this complicated -x-informal stuff?

comment:3 by Ronnie van den Crommenacker, 2 years ago

It's called a private-use subtag extension of the language_tag.
Source: https://en.wikipedia.org/wiki/IETF_language_tag

An optional private-use subtag, composed of the letter x and a hyphen followed by subtags of one to eight characters each, separated by hyphens.

For our use it's because of our language has a more fomal and informal way of communicating. e.g. "Dear <name>" vs "Hi <name>". With the use of the private-use subtag, we can have 2 variants of the same language.

Version 0, edited 2 years ago by Ronnie van den Crommenacker (next)

in reply to:  3 comment:4 by Mariusz Felisiak, 2 years ago

Replying to Ronnie van den Crommenacker:

It's called a private-use subtag extension of the language_tag.
Source: https://en.wikipedia.org/wiki/IETF_language_tag
Source: https://www.w3.org/International/articles/language-tags/#extension

An optional private-use subtag, composed of the letter x and a hyphen followed by subtags of one to eight characters each, separated by hyphens.

makemessages --locale accepts locales in ISO/IEC 15897 format, not IETF language tags. I'm not sure what private subtags are formatted in ISO/IEC 15897 🤔.

comment:5 by Ronnie van den Crommenacker, 2 years ago

I cannot find examples of valid ISO/IEC 15897 private subtags either.

But i found some, non standard locales:
https://www.localeplanet.com/icu/sr-Latn-BA/index.html sr_Latn_BA (this case is mentioned in the django's to_locale comment)
https://www.localeplanet.com/icu/en-US-POSIX/index.html en_US_POSIX

Also in /usr/share/i18n/SUPPORTED i find some other exceptions
ug_CN@latin
sd_IN@devanagari
ca_ES@valencia

So looking at those examples i would suspect the to_locale should return nl_NL_X_INFORMAL or nl_NL_XINFORMAL or nl_Informal_NL or nl_NL@informal

Last edited 2 years ago by Ronnie van den Crommenacker (previous) (diff)

comment:6 by Carlton Gibson, 2 years ago

Hi Ronnie.

The check for - in the locale should only be for the first section a.k.a. nl_NL

Do you have a particular patch in mind? Do you want to make a PR?

I'm not sure reading whether this is really supported or not, but if a fix is small and doesn't break anything, then it's easier to think positively about. (If it's disruptive, the opposite might apply...)

comment:7 by Carlton Gibson, 2 years ago

Resolution: needsinfo
Status: newclosed

I'm going to mark this as needs info. If someone can point to clearer specs, or provide a PoC patch, happy to review… (but pending that 🤷)

comment:8 by Ronnie van den Crommenacker, 2 years ago

Has patch: set
Resolution: needsinfo
Status: closednew

comment:9 by Carlton Gibson, 2 years ago

Hi Ronnie.

Based on the W3.org article, I'd say this isn't something we should really support:

Because these subtags are only meaningful within private agreements and cannot be used interoperably across the Web, they should be used with great care, and avoided whenever possible.

So I would say wontfix.

Then I look at #28546, and see that we did add support in to_locale… — That was your ticket too. I'm guessing you may be the only person using this. 🙂

Can I ask, how come it's taken 5 years for you to hit the issue with makemessages?
Would a project level custom makemessages serve?
This would mean we don't need to add the complication for few (just you?) users 🤔

I have to say I'm still not sure. The patch isn't that big, so we could add it for symmetry with #28546.


in reply to:  9 comment:10 by Ronnie van den Crommenacker, 2 years ago

Based on the W3.org article, I'd say this isn't something we should really support:

Because these subtags are only meaningful within private agreements and cannot be used interoperably across the Web, they should be used with great care, and avoided whenever possible.

So I would say wontfix.

The way i read the comment is "it should be ~used~ with great care", which doesn't not say anything about the ~support~ for the private extension.
It exists in the language-tags for a reason, but for locales it seems not that clear in ISO/IEC 15897 what the restrictions to the format actually are. I have seen some uncommon formats in the wild (see comment https://code.djangoproject.com/ticket/33565?replyto=9#comment:5).

I understand that if something should not be used there is little reason to support it. However, Django has added support for private subtags in the to_locale function already via #28546 and it used to work before Django 3.2.

Then I look at #28546, and see that we did add support in to_locale… — That was your ticket too. I'm guessing you may be the only person using this. 🙂

Yeah, i think i'm the only one. It was (and is) still the perfect solution for the problem we (as a company) are facing. Other solutions would be far more complex.

Can I ask, how come it's taken 5 years for you to hit the issue with makemessages?
Would a project level custom makemessages serve?
This would mean we don't need to add the complication for few (just you?) users 🤔

The makemessages command worked (maybe by accident) perfectly for many years, until we updated to Django 3.2 recently.
This commit included in Django 3.2 added extra checks which, by accident, stopped the support for private subtags
https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

As a temporary solution we rename nl_NL_x_informal to nl_NL-x-informal after we run makemessages

I have to say I'm still not sure. The patch isn't that big, so we could add it for symmetry with #28546.

I hope you will reconsider to include the patch. I'll understand Django wouldn't support it officially, document it and test for it in upcoming releases. If it occasionally breaks, ill understand and i am happy to contribute patches when it breaks again.

comment:11 by Carlton Gibson, 2 years ago

Triage Stage: UnreviewedAccepted

... i am happy to contribute patches when it breaks again.

OK, let's take it for review then :) If no-one objects to the PR, then you can be on the hook to help maintain it. 😄

comment:12 by Mariusz Felisiak, 2 years ago

Owner: changed from nobody to Ronnie van den Crommenacker
Status: newassigned

comment:13 by Carlton Gibson, 22 months ago

Needs tests: set

comment:14 by Carlton Gibson, 22 months ago

Needs tests: unset

Patch is updated. Thanks Ronnie. Unchecking for another review.

comment:15 by Carlton Gibson, 22 months ago

Triage Stage: AcceptedReady for checkin

comment:16 by Carlton Gibson <carlton.gibson@…>, 22 months ago

Resolution: fixed
Status: assignedclosed

In c32858a8:

Fixed #33565 -- Improved locale format validation for the makemessages command.

Note: See TracTickets for help on using tickets.
Back to Top