makemessages fails in UTF-8 mode when processing non-UTF-8 filenames

[Jacob Walls]

Surfaced on CI just now after bumping Windows jobs to Python 3.15, now that UTF-8 mode is the ​default.

ERROR: test_unicode_file_name (i18n.test_extraction.BasicExtractorTests.test_unicode_file_name)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.15.1\x64\Lib\unittest\case.py", line 58, in testPartExecutor
    yield
  File "C:\hostedtoolcache\windows\Python\3.15.1\x64\Lib\unittest\case.py", line 691, in run
    self._callTestMethod(testMethod)
    
  File "C:\hostedtoolcache\windows\Python\3.15.1\x64\Lib\unittest\case.py", line 637, in _callTestMethod
    result = method()
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\tests\i18n\test_extraction.py", line 372, in test_unicode_file_name
    management.call_command("makemessages", locale=[LOCALE], verbosity=0)
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\__init__.py", line 195, in call_command
    return command.execute(*args, **defaults)
      ^^^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\base.py", line 471, in execute
    output = self.handle(*args, **options)
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 423, in handle
    potfiles = self.build_potfiles()
    ^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 504, in build_potfiles
    self.process_files(file_list)
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 596, in process_files
    self.process_locale_dir(locale_dir, files)
    ^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 677, in process_locale_dir
    raise CommandError(
    ^^^^^^^
django.core.management.base.CommandError: errors happened while running xgettext on .\__init__.py
.\ignore_dir\ignored.html.py
.\media_root\media_ignored.html.py
.\static\static_ignored.html.py
.\templates\empty.html.py
.\templates\subdir\ignored.html.py
.\templates\test.html.py
.\templates\xxx_ignored.html.py
.\vidéo.txt.py
xgettext: error while opening ".\vidéo.txt.py" for reading: No such file or directory

I believe the fix is to write using the system locale, which we should have been doing all along (​linter rule on this topic)

django/core/management/commands/makemessages.py

@@ -663 +663 @@
             return
 
         input_files = [bf.work_path for bf in build_files]
-        with NamedTemporaryFile(mode="w+") as input_files_list:
-            input_files_list.write("\n".join(input_files))
+        with NamedTemporaryFile(mode="w+b") as input_files_list:
+            input_files_list.write(
+                "\n".join(input_files).encode(DEFAULT_LOCALE_ENCODING)
+            )
             input_files_list.flush()
             args.extend(["--files-from", input_files_list.name])
             args.extend(self.xgettext_options)

If a triager confirms the approach, I can open the PR. I propose backporting to 6.1 as part of 3.15 support, but with a release note, since it affects released behavior and would have been visible on prior Python versions when running in UTF-8 mode.

[Natalia Bidart]

I was 15 minutes into debugging my PR's failing CI and about to file a ticket when it occurred to me to check for duplicates, and here I am :-) Accepting, of course!

Two notes:

• The failure surfaced after #37396 put gettext on PATH in tests.yml. Until then the test was skipped on Windows (and schedule_tests.yml doesn't install gettext at all), which is why the Windows/Python 3.15 runs stayed green after the bump. I think we should also install gettext in the scheduled tests.
• DEFAULT_LOCALE_ENCODING falls back to "ascii" when locale.getlocale() reports no encoding (for example, the "C" locale, which is (None, None)). In that case .encode() would raise UnicodeEncodeError for non-ASCII names, while today on POSIX it works (the list is written as UTF-8 and xgettext opens the raw bytes). Would it be safer to use os.fsencode() on POSIX, and the ANSI code page (locale.getencoding(), which ignores UTF-8 mode) only on Windows?

References:

• ​https://docs.python.org/3/library/os.html#python-utf-8-mode
• ​https://docs.python.org/3/using/windows.html#win-utf8-mode
• ​https://docs.python.org/3/library/locale.html#locale.getencoding

Proposed patch (commit should also Refs #36664):

.github/workflows/schedule_tests.yml

@@ -34 +34 @@
           python-version: ${{ matrix.python-version }}
           cache: 'pip'
           cache-dependency-path: 'tests/requirements/py3.txt'
+      - name: Install gettext
+        run: |
+          choco install gettext --no-progress -y
+          "C:\Program Files\gettext-iconv\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
       - name: Install and upgrade packaging tools
         run: python -m pip install --upgrade pip wheel
       - run: python -m pip install -r tests/requirements/py3.txt -e .

django/core/management/commands/makemessages.py

@@ -1 +1 @@
 import glob
+import locale
 import os
 import re
 import sys
@@ -663 +664 @@
             return
 
         input_files = [bf.work_path for bf in build_files]
-        with NamedTemporaryFile(mode="w+") as input_files_list:
-            input_files_list.write("\n".join(input_files))
+        # xgettext reads the file list as bytes, decoded with the ANSI code
+        # page on Windows, so it must not depend on Python's UTF-8 mode.
+        if os.name == "nt":
+            files_list = "\n".join(input_files).encode(locale.getencoding())
+        else:
+            files_list = b"\n".join(os.fsencode(path) for path in input_files)
+        with NamedTemporaryFile(mode="w+b") as input_files_list:
+            input_files_list.write(files_list)
             input_files_list.flush()
             args.extend(["--files-from", input_files_list.name])
             args.extend(self.xgettext_options)

[Natalia Bidart]

I agreed with Jacob I will put up a PR for this.

[Natalia Bidart]

Setting as PNI until I see the windows run results.

[nessita <124304+nessita@…>]

In fa214ea7:

Fixed #37447 -- Fixed makemessages with non-ASCII file names on Windows.

Python 3.15 enables UTF-8 mode by default, so the temporary file
listing the files to process was written as UTF-8. xgettext opens
these paths as raw bytes, which Windows interprets in the ANSI code
page, so non-ASCII file names could not be found.

Refs #36664.

[Natalia <124304+nessita@…>]

In be876ef4:

[6.1.x] Fixed #37447 -- Fixed makemessages with non-ASCII file names on Windows.

Python 3.15 enables UTF-8 mode by default, so the temporary file
listing the files to process was written as UTF-8. xgettext opens
these paths as raw bytes, which Windows interprets in the ANSI code
page, so non-ASCII file names could not be found.

Refs #36664.

Backport of fa214ea7b217a37484507b6222cca4915c70b4cc from main.