TranslationCatalog.__getitem__ gives wrong priority to non-plural translations when plural forms mismatch

[UHHHHHHHHHHHHHH]

Description

When a third-party package (e.g. django-allauth) has a slightly different Plural-Forms header than the project's own translations (e.g. plural=n != 1 vs plural=(n != 1)), TranslationCatalog.update() correctly prepends a separate catalog to preserve plural lookup correctness.

However, TranslationCatalog.__getitem__() iterates catalogs in order and returns the first match. This means the prepended catalog wins for all lookups, including non-plural gettext() calls where plural form separation is irrelevant. This silently overrides higher-priority translations (from LOCALE_PATHS or later INSTALLED_APPS) with lower-priority ones.

Steps to reproduce

1. Create a Django project with a translation in LOCALE_PATHS:

   # project/locale/nl/LC_MESSAGES/django.po
   # Plural-Forms: nplurals=2; plural=(n != 1);
   msgid "Activate"
   msgstr "Activeren"
   

2. Install a third-party app (e.g. django-allauth) that defines the same msgid with a different Plural-Forms header:

   # allauth/locale/nl/LC_MESSAGES/django.po  
   # Plural-Forms: nplurals=2; plural=n != 1;
   msgid "Activate"
   msgstr "Activeer"
   

Note: n != 1 and (n != 1) are functionally identical and compile to identical bytecode, but Python's gettext.c2py() produces functions with different __code__ objects. Django's TranslationCatalog.update() compares __code__ to decide whether to merge or prepend.

3. gettext("Activate") returns "Activeer" (allauth's version) instead of "Activeren" (the project's version), even though LOCALE_PATHS should have highest priority per Django's documentation.

Root cause

In django/utils/translation/trans_real.py:

class TranslationCatalog:
    def update(self, trans):
        # Mismatched plural -> prepend to position 0
        for cat, plural in zip(self._catalogs, self._plurals):
            if trans.plural.__code__ == plural.__code__:
                cat.update(trans._catalog)
                break
        else:
            self._catalogs.insert(0, trans._catalog.copy())
            self._plurals.insert(0, trans.plural)

    def __getitem__(self, key):
        # First catalog wins for ALL lookups
        for cat in self._catalogs:
            try:
                return cat[key]
            except KeyError:
                pass
        raise KeyError(key)

The update() prepend is correct for plural lookups (each catalog needs its own plural function). But __getitem__ shouldn't give the prepended catalog priority for non-plural string lookups — these should follow the normal priority order (LOCALE_PATHS > INSTALLED_APPS > Django built-in).

Expected behavior

Non-plural gettext() lookups should respect the documented translation priority order regardless of plural form differences between catalogs.

Actual behavior

A third-party package with a cosmetically different Plural-Forms header silently overrides all matching non-plural translations from higher-priority sources.

Impact

In our project, this causes 28 translations to be silently wrong. The project has no way to fix this without either:

• Monkey-patching TranslationCatalog
• Patching the third-party package's .po files in the Docker build
• Adding pgettext context to every conflicting string

Environment

• Django 5.1 (also affects 5.0, likely all versions since #34221 was fixed)
• Python 3.12
• The issue was introduced/amplified by the fix for #34221 which changed update() to only merge with the top catalog

[Jacob Walls]

I couldn't reproduce:

In [4]: gettext('Activate')
Out[4]: 'Activeren'

After removing my LOCALE_PATHS translations, I get allauth's translation:

In [3]: gettext('Activate')
Out[3]: 'Activeer'

Did you recompile messages?

[UHHHHHHHHHHHHHH]

The reproduction depends on whether the Plural-Forms headers actually mismatch. The bug only triggers when they do.

What allauth ships (nl):

"Plural-Forms: nplurals=2; plural=n != 1;\n"

What Django's own nl translations use:

"Plural-Forms: nplurals=2; plural=(n != 1);\n"

These are functionally identical, but gettext.c2py() produces functions with different __code__ objects:

>>> from gettext import c2py
>>> c2py('(n != 1)').__code__ == c2py('n != 1').__code__
False

If your allauth version's Plural-Forms happens to match Django's (with parentheses), everything merges into a single catalog and LOCALE_PATHS correctly wins. That's likely why you couldn't reproduce.

Here's the exact loading trace on Django 5.1 with allauth 65.15.0:

1. Django's own nl loaded → catalogs = [{django}(P1)] where P1 = (n != 1)
2. allauth loaded → P2 = n != 1, iterates catalogs, P2.__code__ != P1.__code__ → prepended → catalogs = [{allauth}(P2), {django}(P1)]
3. LOCALE_PATHS loaded → P1 = (n != 1), iterates catalogs: doesn't match P2 at position 0, matches P1 at position 1 → merged into Django's catalog → catalogs = [{allauth}(P2), {django+project}(P1)]

__getitem__() iterates in order → allauth at position 0 wins for all shared msgids.

Minimal reproducer (4 files):

requirements.txt:

django>=5.1,<5.2
django-allauth>=65.0

settings.py:

import os

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

SECRET_KEY = "not-secret"
USE_I18N = True
LANGUAGE_CODE = "nl"

LOCALE_PATHS = [os.path.join(BASE_DIR, "locale")]

INSTALLED_APPS = [
    "allauth",
]

locale/nl/LC_MESSAGES/django.po (compile with msgfmt -o django.mo django.po):

# Project translations — should have highest priority via LOCALE_PATHS
msgid ""
msgstr ""
"Language: nl\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

msgid "Incorrect password."
msgstr "Wachtwoord onjuist (PROJECT OVERRIDE)."

reproduce.py:

import os
import django.conf

django.conf.settings.configure(
    USE_I18N=True,
    LANGUAGE_CODE="nl",
    LOCALE_PATHS=[os.path.join(os.path.dirname(__file__), "locale")],
    INSTALLED_APPS=["allauth"],
)

import django
django.setup()

from django.utils.translation import activate, gettext
from django.utils.translation.trans_real import _translations

activate("nl")

trans = _translations.get("nl")
if trans:
    print(f"Number of catalogs: {len(trans._catalog._catalogs)}")
    for i, (cat, plural) in enumerate(zip(trans._catalog._catalogs, trans._catalog._plurals)):
        print(f"\n  Catalog {i}: {len(cat)} entries")
        if "Incorrect password." in cat:
            print(f"    'Incorrect password.' -> {cat['Incorrect password.']!r}")

result = gettext("Incorrect password.")
expected = "Wachtwoord onjuist (PROJECT OVERRIDE)."

print(f"\ngettext('Incorrect password.') = {result!r}")

if result == expected:
    print("\nOK: LOCALE_PATHS translation wins (correct behavior)")
else:
    print(f"\nBUG: Expected LOCALE_PATHS: {expected!r}")
    print(f"     Got allauth's version: {result!r}")

Output:

Number of catalogs: 2

  Catalog 0: 377 entries
    'Incorrect password.' -> 'Ongeldig wachtwoord.'

  Catalog 1: 365 entries
    'Incorrect password.' -> 'Wachtwoord onjuist (PROJECT OVERRIDE).'

gettext('Incorrect password.') = 'Ongeldig wachtwoord.'

BUG: Expected LOCALE_PATHS: 'Wachtwoord onjuist (PROJECT OVERRIDE).'
     Got allauth's version: 'Ongeldig wachtwoord.'

Side note: In Django 5.2, update() was changed to only compare with _plurals[0] (the top catalog) instead of iterating all catalogs. This accidentally masks this specific scenario — LOCALE_PATHS also gets prepended and lands at position 0, so the project wins. But the underlying design issue (that __getitem__ gives prepended catalogs priority for non-plural lookups) remains.

[Jacob Walls]

Oh, so it affects Django 5.1 only. That's why it worked for me. 5.1 doesn't receive bug fixes any more. Thanks for clarifying.