Broken translations since the introduction of TranslationCatalog

[Hristo Gatsinski]

I recently updated one of my projects from the latest patch version of 2.2 to 3.1 and noticed that one of the translations is broken. When I translate "All" to Bulgarian it should output "Всичко" but instead, I get "Все" which I believe is in Russian. The PO files in Django are correct, and I have the same string translated in my project too. Both with the correct msgstr.

I managed to track the issue to the introduction of TranslationCatalog in 3.0.5 in this commit:

​https://github.com/django/django/commit/d9f1792c7649e9f946f4a3a35a76bddf5a412b8b

Replacing _catalog from TranslationCatalog() to normal Python dict (as it used to be in 3.0.4 and prior) resolves the issue.

The issue continues to exist in the latest 3.1 and the pre-release 3.2a1.

I use Linux Mint 20.1 based on Ubuntu 20.04. For a while I suspected gettext. The last version available via apt is 0.19.8.1 but I also installed the latest from source (0.21). The issue still exists.

The bug is easy to reproduce.

Enter the shell of any project with Django 3.0.4 and older and execute the following:

from django.utils import translation
translation.activate('bg')
translation.gettext('All')

You will see "Всичко".

Do the same with Django 3.0.5 and above and you will get "Все" which is an incorrect translation. It doesn't matter if you are using gettext, ugettext, ugettext_lazy, etc.

I failed to find an existing ticket describing this problem. Forgive me if there is one already.

[Carlton Gibson]

Hi Hristo

Thanks for the report. Can I ask you to dig a bit more since I can't reproduce as described.

Here's a full transcript showing, creation of a project with Django 3.0.4, with the expected result, but the same output for 3.0.5 and 3.1.5:

~ $ mktmpenv 
created virtual environment CPython3.8.3.final.0-64 in 865ms
...
This is a temporary environment. It will be deleted when you run 'deactivate'.
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8 $ pip install Django==3.0.4
Collecting Django==3.0.4
  Using cached Django-3.0.4-py3-none-any.whl (7.5 MB)
Collecting asgiref~=3.2
  Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
Collecting pytz
  Using cached pytz-2020.5-py2.py3-none-any.whl (510 kB)
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.1-py3-none-any.whl (42 kB)
Installing collected packages: sqlparse, pytz, asgiref, Django
Successfully installed Django-3.0.4 asgiref-3.3.1 pytz-2020.5 sqlparse-0.4.1
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8 $ django-admin startproject issue32373
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8 $ cd issue32373/
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8/issue32373 $ ./manage.py shell
Python 3.8.3 (default, Jul 13 2020, 16:58:09) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.utils import translation
>>> translation.activate('bg')
>>> translation.gettext('All')
'Всички'
>>> 
now exiting InteractiveConsole...
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8/issue32373 $ pip install Django==3.0.5
Collecting Django==3.0.5
  Using cached Django-3.0.5-py3-none-any.whl (7.5 MB)
Requirement already satisfied: pytz in /Users/carlton/ve/tmp-1330abac7a589b8/lib/python3.8/site-packages (from Django==3.0.5) (2020.5)
Requirement already satisfied: asgiref~=3.2 in /Users/carlton/ve/tmp-1330abac7a589b8/lib/python3.8/site-packages (from Django==3.0.5) (3.3.1)
Requirement already satisfied: sqlparse>=0.2.2 in /Users/carlton/ve/tmp-1330abac7a589b8/lib/python3.8/site-packages (from Django==3.0.5) (0.4.1)
Installing collected packages: Django
  Attempting uninstall: Django
    Found existing installation: Django 3.0.4
    Uninstalling Django-3.0.4:
      Successfully uninstalled Django-3.0.4
Successfully installed Django-3.0.5
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8/issue32373 $ ./manage.py shell
Python 3.8.3 (default, Jul 13 2020, 16:58:09) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.utils import translation
>>> translation.activate('bg')
>>> translation.gettext('All')
'Всички'
>>> import django
>>> django.VERSION
(3, 0, 5, 'final', 0)
>>> 
now exiting InteractiveConsole...
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8/issue32373 $ pip install -U Django
Requirement already satisfied: Django in /Users/carlton/ve/tmp-1330abac7a589b8/lib/python3.8/site-packages (3.0.5)
Collecting Django
  Using cached Django-3.1.5-py3-none-any.whl (7.8 MB)
Requirement already satisfied: pytz in /Users/carlton/ve/tmp-1330abac7a589b8/lib/python3.8/site-packages (from Django) (2020.5)
Requirement already satisfied: asgiref<4,>=3.2.10 in /Users/carlton/ve/tmp-1330abac7a589b8/lib/python3.8/site-packages (from Django) (3.3.1)
Requirement already satisfied: sqlparse>=0.2.2 in /Users/carlton/ve/tmp-1330abac7a589b8/lib/python3.8/site-packages (from Django) (0.4.1)
Installing collected packages: Django
  Attempting uninstall: Django
    Found existing installation: Django 3.0.5
    Uninstalling Django-3.0.5:
      Successfully uninstalled Django-3.0.5
Successfully installed Django-3.1.5
(tmp-1330abac7a589b8) ~/ve/tmp-1330abac7a589b8/issue32373 $ ./manage.py shell
Python 3.8.3 (default, Jul 13 2020, 16:58:09) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.utils import translation
>>> translation.activate('bg')
>>> translation.gettext('All')
'Всички'
>>> 

Really very happy to look into the issue here (no problem with the new ticket! :) but what have I done wrong to not reproduce it?
Thanks!

[Hristo Gatsinski]

Hi, Carlton.

I'm so sorry for the trouble. It was getting late yesterday I didn't bother to check if the issue exists in a fresh project. And it doesn't. I started investigating it against my project and I found out that the cause is not Django but django-filter. Before opening the ticket I tested my third party libraries by downgrading them to the version prior to the update but somehow I missed it. I also had to downgrade Django and maybe this is how I got confused. Or maybe I just really wanted to find a problem within Django and feel good for helping the community. Well, it didn't turn out as I expected.

I created a fresh project with only django-filter installed, with a simple test and CircleCI integration to verify the problem.

The repository is available here - ​https://github.com/gatsinski/django-filters-translation-issue
The failing CI job can be found here - ​https://app.circleci.com/pipelines/github/gatsinski/django-filters-translation-issue/3/workflows/4e5e56b0-39a2-4f74-888e-e1fa0c7e090c/jobs/3

Tonight I will check the docs of django-filter and browse the changes to see if there is something related between 2.2 and 2.4 and if not, I will open another ticket there.

And once again, sorry for the trouble. I shouldn't open tickets that late. Forget about this one and we will probably speak again on django-filter's GitHub repo.