#37447 assigned Bug

makemessages fails in UTF-8 mode when processing non-UTF-8 filenames

Reported by: Jacob Walls Owned by: Jacob Walls
Component: Internationalization Version: 6.1
Severity: Release blocker Keywords: makemessages, locale, utf-8
Cc: Sarah Boyce Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Surfaced on CI just now after bumping Windows jobs to Python 3.15, now that UTF-8 mode is the ​default.

ERROR: test_unicode_file_name (i18n.test_extraction.BasicExtractorTests.test_unicode_file_name)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.15.1\x64\Lib\unittest\case.py", line 58, in testPartExecutor
    yield
  File "C:\hostedtoolcache\windows\Python\3.15.1\x64\Lib\unittest\case.py", line 691, in run
    self._callTestMethod(testMethod)
    
  File "C:\hostedtoolcache\windows\Python\3.15.1\x64\Lib\unittest\case.py", line 637, in _callTestMethod
    result = method()
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\tests\i18n\test_extraction.py", line 372, in test_unicode_file_name
    management.call_command("makemessages", locale=[LOCALE], verbosity=0)
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\__init__.py", line 195, in call_command
    return command.execute(*args, **defaults)
      ^^^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\base.py", line 471, in execute
    output = self.handle(*args, **options)
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 423, in handle
    potfiles = self.build_potfiles()
    ^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 504, in build_potfiles
    self.process_files(file_list)
    ^^^^^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 596, in process_files
    self.process_locale_dir(locale_dir, files)
    ^^^^^^^^^^^
  File "D:\a\django\django\django\core\management\commands\makemessages.py", line 677, in process_locale_dir
    raise CommandError(
    ^^^^^^^
django.core.management.base.CommandError: errors happened while running xgettext on .\__init__.py
.\ignore_dir\ignored.html.py
.\media_root\media_ignored.html.py
.\static\static_ignored.html.py
.\templates\empty.html.py
.\templates\subdir\ignored.html.py
.\templates\test.html.py
.\templates\xxx_ignored.html.py
.\vidéo.txt.py
xgettext: error while opening ".\vidéo.txt.py" for reading: No such file or directory

I believe the fix is to write using the system locale, which we should have been doing all along (​linter rule on this topic)

  • django/core/management/commands/makemessages.py

    diff --git a/django/core/management/commands/makemessages.py b/django/core/management/commands/makemessages.py
    index 5a4bd55bb8..9db5a6ef2a 100644
    a b class Command(BaseCommand):  
    663663            return
    664664
    665665        input_files = [bf.work_path for bf in build_files]
    666         with NamedTemporaryFile(mode="w+") as input_files_list:
    667             input_files_list.write("\n".join(input_files))
     666        with NamedTemporaryFile(mode="w+b") as input_files_list:
     667            input_files_list.write(
     668                "\n".join(input_files).encode(DEFAULT_LOCALE_ENCODING)
     669            )
    668670            input_files_list.flush()
    669671            args.extend(["--files-from", input_files_list.name])
    670672            args.extend(self.xgettext_options)

If a triager confirms the approach, I can open the PR. I propose backporting to 6.1 as part of 3.15 support, but with a release note, since it affects released behavior and would have been visible on prior Python versions when running in UTF-8 mode.

Change History (0)

Note: See TracTickets for help on using tickets.
Back to Top