Opened 4 years ago

Closed 4 years ago

#32762 closed Bug (fixed)

test_msgfmt_error_including_non_ascii fails on Mac OS Catalina, localized msgfmt binary.

Reported by: Nilo César Teixeira Owned by: Nilo César Teixeira
Component: Internationalization Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

FAIL: test_msgfmt_error_including_non_ascii (i18n.test_compilation.CompilationErrorHandling)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/teixeira/Downloads/Pessoal/etudes/etudes_py/django_source/tests/i18n/test_compilation.py", line 204, in test_msgfmt_error_including_non_ascii
    self.assertIn("' cannot start a field name", stderr.getvalue())
AssertionError: "' cannot start a field name" not found in 'Execution of msgfmt failed: /private/var/folders/sj/kll9fdms3c52l7m19_f_zn980000gn/T/django_al6dgjzh/i18n_cam5a1xi/commands/locale/ko/LC_MESSAGES/django.po:24: "msgstr" não é um formato de string Python brace válido, ao contrário de "msgid". Motivo: Na diretiva número 0, "�" não pode iniciar um nome de campo.\nmsgfmt: encontrado 1 erro fatal\n'

The assertIn fails because the msgfmt output is in pt-BR. The export LANG=C environment setting on this test doesn't help changing the binary msgfmt output to English.

Change History (14)

comment:1 by Nilo César Teixeira, 4 years ago

I found a fix!

env.update({'LC_ALL': 'C'})

This worked!

comment:2 by Carlton Gibson, 4 years ago

Resolution: invalid
Status: newclosed

Hi.

I'm going to close this as Invalid because I'm pretty sure it's a system configuration issue.

Happy to help if I can…

In a new shell try the locale command. You basically want something claiming to be UTF-8:

$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

You want pt-br right? So putting export LC_ALL=pt_BR.UTF-8 in your shell startup script may/should solve your issues.

comment:3 by Claude Paroz, 4 years ago

Carlton, I'm not sure you got the idea of the ticket. The problem is that we are checking an English error message content in the Django test CompilationErrorHandling.test_msgfmt_error_including_non_ascii, and we try to force unlocalize message by env.update({'LANG': 'C'}) which looks like is not sufficient in certain systems.

comment:4 by Carlton Gibson, 4 years ago

Resolution: invalid
Status: closednew

comment:5 by Carlton Gibson, 4 years ago

OK, let's have another look, thanks Claude.

...which looks like is not sufficient in certain systems.

Not sure how to proceed.
I can't reproduce without more details on macOS Big Sur.
I'm afraid I don't have Catalina available (but this test never failed for me over the last few years.)

It still looks like a locale issue 🤔 @niloct what output do you get there?

comment:6 by Nilo César Teixeira, 4 years ago

~/Downloads/Pessoal/etudes/etudes_py/django_source/tests $ locale
LANG="pt_BR.UTF-8"
LC_COLLATE="pt_BR.UTF-8"
LC_CTYPE="pt_BR.UTF-8"
LC_MESSAGES="pt_BR.UTF-8"
LC_MONETARY="pt_BR.UTF-8"
LC_NUMERIC="pt_BR.UTF-8"
LC_TIME="pt_BR.UTF-8"
LC_ALL="pt_BR.UTF-8"

For some reason LANG isn't enough to reset locale settings, but LC_ALL was.

If I change the environment override on the aforementioned test to LC_ALL instead of LANG, the test passes.

Last edited 4 years ago by Nilo César Teixeira (previous) (diff)

comment:7 by Carlton Gibson, 4 years ago

Triage Stage: UnreviewedAccepted

Super, thanks for the extra info.

$ LC_ALL="pt_BR.UTF-8" ./runtests.py i18n

Is sufficient to cause the failure. So the env update will fix it, as you say.

diff --git a/tests/i18n/test_compilation.py b/tests/i18n/test_compilation.py
index 791c1d4f15..915d65e6e1 100644
--- a/tests/i18n/test_compilation.py
+++ b/tests/i18n/test_compilation.py
@@ -193,7 +193,7 @@ class CompilationErrorHandling(MessageCompilationTests):
         # po file contains invalid msgstr content (triggers non-ascii error content).
         # Make sure the output of msgfmt is unaffected by the current locale.
         env = os.environ.copy()
-        env.update({'LANG': 'C'})
+        env.update({'LANG': 'C', 'LC_ALL': 'C'})
         with mock.patch('django.core.management.utils.run', lambda *args, **kwargs: run(*args, env=env, **kwargs)):
             cmd = MakeMessagesCommand()
             if cmd.gettext_version < (0, 18, 3):

Would you like to make a PR @niloct?

comment:8 by Carlton Gibson, 4 years ago

Checking the locale man page, LC_ALL if present trumps all else, so the LANG setting can be replaced in the test, rather than added to, I think.

in reply to:  8 comment:9 by Nilo César Teixeira, 4 years ago

Replying to Carlton Gibson:

Checking the locale man page, LC_ALL if present trumps all else, so the LANG setting can be replaced in the test, rather than added to, I think.

Hi Carlton!

Yes I saw that man. Awesome that you concur :)

Sorry for the delay in response, I wasn't notified because I hadn't updated my profile with my e-mail, despite the big yellow box shouting this to me!

Regarding the pull request, I'm just starting reading through the docs (finished the toast contribution), would I have to change any docs in the pr or just the test with a comment about it ?

Thanks!

Version 0, edited 4 years ago by Nilo César Teixeira (next)

comment:10 by Nilo César Teixeira, 4 years ago

Owner: changed from nobody to Nilo César Teixeira
Status: newassigned

comment:11 by Nilo César Teixeira, 4 years ago

Has patch: set

Well, I did it I guess.

Please take a look. It's my first contribution to Django :)

https://github.com/django/django/pull/14426

comment:12 by Carlton Gibson, 4 years ago

Super, thanks Nilo — Welcome aboard! :) — I'll take a look.

comment:13 by Carlton Gibson, 4 years ago

Triage Stage: AcceptedReady for checkin

comment:14 by Carlton Gibson <carlton.gibson@…>, 4 years ago

Resolution: fixed
Status: assignedclosed

In 0d67481:

Fixed #32762 -- Fixed locale reset in compilemessages test.

Reset the LC_ALL override value in the test environment to ensure that locale
values the calling environment are not used.

Note: See TracTickets for help on using tickets.
Back to Top