id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
19919	"get_language_from_request() disregards ""en-us"" and ""en"" languages when matching Accept-Language header"	lanzz@…	Łukasz Langa	"Scenario:
Django settings with languages:
{{{
LANGUAGES = (
    ('bg', gettext('Bulgarian')),
    ('en-us', gettext('English')),
)
}}}
A request comes with header:
{{{
Accept-Language: en-US,en;q=0.8,bg;q=0.6,ru;q=0.4
}}}

Without a previously set language in the session or a cookie, {{{get_language_from_request()}}} should inspect the header and pick the ""en-us"" language, as it is of the highest priority. Now, I see that {{{get_language_from_request()}}} actually disregards the weight, but still it should pick the ""en-us"" language, as it is listed first in the header, and is an explicitly enabled language in the settings. This is hot what happens, however.

{{{get_language_from_request()}}} first normalizes each accepted language to its locale name, as mapped by Python's {{{locale.locale_alias}}}. This turns both ""en-US"" and ""en"" into the canonical locale name ""en_US"". Then it tests if either ""en_US"" or its primary component ""en"" is a supported language. It scans over {{{all_locale_paths()}}} and tests if any of them contains an {{{en_US/LC_MESSAGES/django.mo}}} file, which none does, because Django does not provide an ""en_US"" translation, thus the ""en_US"" locale is rejected. Then it sees that ""en"" is not listed in {{{LANGUAGES}}}, and rejects that as well. It ends up returning ""bg"" as detected language, even though it is of a lower priority AND listed later than both ""en-US"" and ""en"" in the header.

Since the ""en-us"" language is the default for the {{{LANGUAGE_CODE}}} setting, and there are cases where Django forces the language to ""en-us"", it seems very unexpected to me that it is not possible to use that language code in {{{LANGUAGES}}} without it being completely rejected during {{{Accept-Language}}} matching. I am forced into using ""en"" in my {{{LANGUAGES}}}, even though my management commands, for instance, will run under a forced ""en-us"" language (which would be technically unsupported as per the {{{LANGUAGES}}} setting).

It seems to me the problem in this case is [https://github.com/django/django/blob/master/django/utils/translation/trans_real.py#L443 line 443] in {{{django/utils/translation/trans_real.py}}}. At that point it would check if ""en"" is in {{{LANGUAGES}}}, even though we're actually processing the ""en-US"" entry from the {{{Accept-Language}}} header. It makes no sense to reject ""en-US"" because ""en"" is not in {{{LANGUAGES}}}, even though ""en-US"" itself IS listed there.

(Note that this has been tested with 1.5c2, though it is not an option in the Version select box here)"	Bug	closed	Internationalization	1.5-rc-1	Normal	fixed	localization		Accepted	0	0	1	0	0	0