get_language_from_request() disregards "en-us" and "en" languages when matching Accept-Language header

[lanzz@…]

Scenario:
Django settings with languages:

LANGUAGES = (
    ('bg', gettext('Bulgarian')),
    ('en-us', gettext('English')),
)

A request comes with header:

Accept-Language: en-US,en;q=0.8,bg;q=0.6,ru;q=0.4

Without a previously set language in the session or a cookie, get_language_from_request() should inspect the header and pick the "en-us" language, as it is of the highest priority. Now, I see that get_language_from_request() actually disregards the weight, but still it should pick the "en-us" language, as it is listed first in the header, and is an explicitly enabled language in the settings. This is hot what happens, however.

get_language_from_request() first normalizes each accepted language to its locale name, as mapped by Python's locale.locale_alias. This turns both "en-US" and "en" into the canonical locale name "en_US". Then it tests if either "en_US" or its primary component "en" is a supported language. It scans over all_locale_paths() and tests if any of them contains an en_US/LC_MESSAGES/django.mo file, which none does, because Django does not provide an "en_US" translation, thus the "en_US" locale is rejected. Then it sees that "en" is not listed in LANGUAGES, and rejects that as well. It ends up returning "bg" as detected language, even though it is of a lower priority AND listed later than both "en-US" and "en" in the header.

Since the "en-us" language is the default for the LANGUAGE_CODE setting, and there are cases where Django forces the language to "en-us", it seems very unexpected to me that it is not possible to use that language code in LANGUAGES without it being completely rejected during Accept-Language matching. I am forced into using "en" in my LANGUAGES, even though my management commands, for instance, will run under a forced "en-us" language (which would be technically unsupported as per the LANGUAGES setting).

It seems to me the problem in this case is ​line 443 in django/utils/translation/trans_real.py. At that point it would check if "en" is in LANGUAGES, even though we're actually processing the "en-US" entry from the Accept-Language header. It makes no sense to reject "en-US" because "en" is not in LANGUAGES, even though "en-US" itself IS listed there.

(Note that this has been tested with 1.5c2, though it is not an option in the Version select box here)

[Łukasz Langa <lukasz@…>]

In 92ebb29c5376c1811e8607e96bddc90d24e44e2a:

Fixes #19919: get_language_from_request() disregards "en-us" and "en" languages
when matching Accept-Language

[Aymeric Augustin <aymeric.augustin@…>]

In cac7b44571b497bfecad2b786ae7c6172aed842e:

Merge pull request #1100 from ambv/issue19919

Fixed #19919: get_language_from_request() disregards en-us and en