Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#19919 closed Bug (fixed)

get_language_from_request() disregards "en-us" and "en" languages when matching Accept-Language header

Reported by: lanzz@… Owned by: ambv
Component: Internationalization Version: 1.5-rc-1
Severity: Normal Keywords: localization
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: yes Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Scenario:
Django settings with languages:

LANGUAGES = (
    ('bg', gettext('Bulgarian')),
    ('en-us', gettext('English')),
)

A request comes with header:

Accept-Language: en-US,en;q=0.8,bg;q=0.6,ru;q=0.4

Without a previously set language in the session or a cookie, get_language_from_request() should inspect the header and pick the "en-us" language, as it is of the highest priority. Now, I see that get_language_from_request() actually disregards the weight, but still it should pick the "en-us" language, as it is listed first in the header, and is an explicitly enabled language in the settings. This is hot what happens, however.

get_language_from_request() first normalizes each accepted language to its locale name, as mapped by Python's locale.locale_alias. This turns both "en-US" and "en" into the canonical locale name "en_US". Then it tests if either "en_US" or its primary component "en" is a supported language. It scans over all_locale_paths() and tests if any of them contains an en_US/LC_MESSAGES/django.mo file, which none does, because Django does not provide an "en_US" translation, thus the "en_US" locale is rejected. Then it sees that "en" is not listed in LANGUAGES, and rejects that as well. It ends up returning "bg" as detected language, even though it is of a lower priority AND listed later than both "en-US" and "en" in the header.

Since the "en-us" language is the default for the LANGUAGE_CODE setting, and there are cases where Django forces the language to "en-us", it seems very unexpected to me that it is not possible to use that language code in LANGUAGES without it being completely rejected during Accept-Language matching. I am forced into using "en" in my LANGUAGES, even though my management commands, for instance, will run under a forced "en-us" language (which would be technically unsupported as per the LANGUAGES setting).

It seems to me the problem in this case is line 443 in django/utils/translation/trans_real.py. At that point it would check if "en" is in LANGUAGES, even though we're actually processing the "en-US" entry from the Accept-Language header. It makes no sense to reject "en-US" because "en" is not in LANGUAGES, even though "en-US" itself IS listed there.

(Note that this has been tested with 1.5c2, though it is not an option in the Version select box here)

Change History (4)

comment:1 Changed 2 years ago by ambv

  • Needs documentation unset
  • Needs tests set
  • Owner changed from nobody to ambv
  • Patch needs improvement unset
  • Status changed from new to assigned
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 2 years ago by aaugustin

  • Component changed from Uncategorized to Internationalization

comment:3 Changed 2 years ago by Łukasz Langa <lukasz@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In 92ebb29c5376c1811e8607e96bddc90d24e44e2a:

Fixes #19919: get_language_from_request() disregards "en-us" and "en" languages
when matching Accept-Language

comment:4 Changed 2 years ago by Aymeric Augustin <aymeric.augustin@…>

In cac7b44571b497bfecad2b786ae7c6172aed842e:

Merge pull request #1100 from ambv/issue19919

Fixed #19919: get_language_from_request() disregards en-us and en

Note: See TracTickets for help on using tickets.
Back to Top