Translatable URL patterns raise 404 for non-English default language when prefix_default_language=False is used.

[ab]

A simple django project with instruction to replicate the bug can be found here:
​github repo

In brief: prefix_default_language = False raises HTTP 404 for the default unprefixed pages if LANGUAGE_CODE is not "en".

I think the problem is that the function get_language_from_path in django/utils/translation/trans_real.py returns None in case of failure instead of LANGUAGE_CODE: ​diff in 4.2

Consequently, other mechanisms are used to get the language (cookies or headers) that do not work neither.

Related issue with my last comment adding some extra context: https://code.djangoproject.com/ticket/34455

It is the first time I contribute to django, I hope the bug report is OK. I am also willing to write the patch and test if required.

[ab]

Expected behavior: ​django 4.2 documentation

LocaleMiddleware tries to determine the user’s language preference by following this algorithm:

• First, it looks for the language prefix in the requested URL. This is only performed when you are using the i18n_patterns function in your root URLconf. See Internationalization: in URL patterns for more information about the language prefix and how to internationalize URL patterns.

• Failing that, it looks for a cookie. The name of the cookie used is set by the LANGUAGE_COOKIE_NAME setting. (The default name is django_language.)

• Failing that, it looks at the Accept-Language HTTP header. This header is sent by your browser and tells the server which language(s) you prefer, in order by priority. Django tries each language in the header until it finds one with available translations.

• Failing that, it uses the global LANGUAGE_CODE setting.

[Mariusz Felisiak]

Thanks for the report. The use of URL patterns marked as translatable is crucial for this bug.

Regression in 94e7f471c4edef845a4fe5e3160132997b4cca81.
Reproduced at c24cd6575f948661fa0ed8b27b79098610dc3ccc.

[Mariusz Felisiak]

Replying to ab:

Expected behavior: ​django 4.2 documentation

LocaleMiddleware tries to determine the user’s language preference by following this algorithm:

• First, it looks for the language prefix in the requested URL. This is only performed when you are using the i18n_patterns function in your root URLconf. See Internationalization: in URL patterns for more information about the language prefix and how to internationalize URL patterns.

• Failing that, it looks for a cookie. The name of the cookie used is set by the LANGUAGE_COOKIE_NAME setting. (The default name is django_language.)

• Failing that, it looks at the Accept-Language HTTP header. This header is sent by your browser and tells the server which language(s) you prefer, in order by priority. Django tries each language in the header until it finds one with available translations.

• Failing that, it uses the global LANGUAGE_CODE setting.

IMO it still works that way. However, in Django 4.2 get_language_from_request() returns the language from a request (en for me) which is activated and the default path about/ is no longer translated to the a-propos/. This is definitely a change from the previous behavior.

[ab]

Replying to Mariusz Felisiak:

IMO it still works that way. However, in Django 4.2 get_language_from_request() returns the language from a request (en for me) which is activated and the default path about/ is no longer translated to the a-propos/. This is definitely a change from the previous behavior.

Thank you Mariusz for the quick reaction.

I agree it still globally works that way, nevertheless, in the case I describe, when django looks for the language prefix in the requested URL and there is not language prefix, I would expect django to return "fr", not to go to the next steps of the algorithm. Because I want prefix_default_language = False to take precedence on cookies or headers. Does it make sense?

I need to add that I use translate_url to ​build the links in my templates.
Consequently, my URLs are translated in the template only (hence the 404).

So you're right about the default path not being translated anymore.

[Sarah Boyce]

I have a PR with what I think the issue is, but not confident ​https://github.com/django/django/pull/16797
@ab I think what you're saying makes sense

[David Sanders]

I agree it still globally works that way, nevertheless, in the case I describe, when django looks for the language prefix in the requested URL and there is not language prefix, I would expect django to return "fr", not to go to the next steps of the algorithm. Because I want prefix_default_language = False to take precedence on cookies or headers. Does it make sense?

My 2¢: Is ignoring the process for determining the language the least surprising choice here though? It all depends on whether no-prefix URL should refer to a user setting or the site's default language. I mean imho navigating to a URL I might expect it to show the language I chose 🤷‍♂️

[ab]

@Sarah: yes, it is the same problem. After investigating the code, the change in behavior is linked to the fact that get_language_from_path returns None when the url is not prefixed. So, the cookie is used or the Accept-Language header sent by the browser. In my case, I think it is the HTTP header.

@David: thanks for your contribution, but I do not fully agree. If prefix_default_url is True, the language is correctly detected by django based on the URL. If I set prefix_default_url to False I expect the same behavior for the default language without prefix. When I decide do use i18n_patterns at least (I have just added this tag to the ticket).
When i18n_patternsis not used, I agree with you.

So the problem might come from i18n_patterns not calling/handling correctly the calls to the new get_language_* functions.

[ab]

@sarah: I'll test your patch because your edits might solve the problem with HTTP headers too. Thanks!

[ab]

Just to keep track of the current work on this issue, there is a discussion about how django should behave here: ​https://github.com/django/django/pull/16797#issuecomment-1524958085

As suggested by Sarah, I'll post to django-developers for a wider range of opinions.

[Mariusz Felisiak]

​Django forum discussion.

[Mariusz Felisiak]

Alternative ​PR.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 0e444e8:

Fixed #34515 -- Made LocaleMiddleware prefer language from paths when i18n patterns are used.

Regression in 94e7f471c4edef845a4fe5e3160132997b4cca81.

This reverts commit 94e7f471c4edef845a4fe5e3160132997b4cca81
(refs #34069) and
partly reverts commit 3b4728310a7a64f8fcc548163b0aa5f98a5c78f5.

Thanks Anthony Baillard for the report.

Co-Authored-By: Sarah Boyce <42296566+sarahboyce@…>

[Mariusz Felisiak <felisiak.mariusz@…>]

In f200d836:

[4.2.x] Fixed #34515 -- Made LocaleMiddleware prefer language from paths when i18n patterns are used.

Regression in 94e7f471c4edef845a4fe5e3160132997b4cca81.

This reverts commit 94e7f471c4edef845a4fe5e3160132997b4cca81
(refs #34069) and
partly reverts commit 3b4728310a7a64f8fcc548163b0aa5f98a5c78f5.

Thanks Anthony Baillard for the report.

Co-Authored-By: Sarah Boyce <42296566+sarahboyce@…>

Backport of 0e444e84f87d174713a2aef0c4f9704ce2865586 from main

[ab]

Thank you for working on this and maintaining django so well!

Anthony

[Foucauld Degeorges]

Quite logically, this fix recreated the previous bug https://code.djangoproject.com/ticket/32886.

A summary of my understanding of the issue:

The root cause seems to be that in an app where the following routes coexist:

• type A: a language-independant route. The language cannot be inferred from calling such routes.
• type B: a language-dependant route, with a prefix from i18n_patterns, with prefix_default_language=False. The language can be inferred from calling such routes, and will supersede other sources of language.

LocaleMiddleware cannot tell those apart. With the code currently on master, LocaleMiddleware detects that at least one URL of type B exists in the app, and will assume that all routes of type A or B are of type B. It will always choose to infer the language.

If I understand correctly, the PR that fixed that other bug, simply inverted the behaviour and created this bug here. Reverting that PR recreated the old bug.

The actual fix would be for LocaleMiddleware to be able to know whether the specific route is of type A or B, and based on that, choose to infer the language of the request or not.
As a workaround, we have implemented our own LocaleMiddleware which:

• uses logic adapted from django-extensions' show_urls command to list all routes in all enabled languages
• groups them by name to see whether some routes have more than one distinct pattern
• this tells us whether a route is translated (of type B), or not translated (of type A)
• if type B, LocaleMiddleware should infer the language from the route. If type A, it shouldn't.

edit: I'll post this again on the other bug report, that's the one that's still present.