makemessages command doesn't set .pot file charset properly

[Bartosz Grabski]

When running python manage.py makemessages, I'm getting the following error:

CommandError: errors happened while running msguniq
C:\dev\xxx\locale\django.pot:2738: C:\dev\xxx\locale\django.pot: input is not valid in "ASCII" encoding

This is because some of my translatable strings contain non-ASCII characters. I've checked the code in makemessages.py and found the culprit:

        for line in pot_lines:
            if not found and not header_read:
                found = True
                line = line.replace('charset=CHARSET', 'charset=UTF-8')
            if not line and not found:
                header_read = True
            lines.append(line)

Since found is set to True on the first iteration, charset is never updated as it's usually on line 17.

[Bartosz Grabski]

PR: ​https://github.com/django/django/pull/9997

[Ingo Klöcker]

I would add a test for this.

[Bartosz Grabski]

Will do.

[Ramiro Morales]

I'm the one who introduced this bug in 6ab0d1358fc78077064aab88a4fb0a47ca116391. Mea culpa.

I can contribute a test case (also attached):

tests/i18n/commands/templates/test.html

@@ -105 +105 @@
 {% endblocktrans %}
 
 {% trans "Non-breaking space :" %}
+
+{% trans "Nón-ÁSCÍÏ text" %}

tests/i18n/test_extraction.py

@@ -394 +394 @@
             po_contents = fp.read()
             self.assertMsgStr("Größe", po_contents)
 
+    def test_pot_charset_header_is_utf8(self):
+        self.assertFalse(os.path.exists(self.POT_FILE))
+        management.call_command('makemessages', locale=[LOCALE], verbosity=0, keep_pot=True)
+        self.assertTrue(os.path.exists(self.POT_FILE))
+        with open(self.POT_FILE, 'r', encoding='utf-8') as fp:
+            contents = fp.read()
+            self.assertIn(r'; charset=UTF-8\n"', contents)
+
 
 class JavascriptExtractorTests(ExtractorTests):
 

Problem is I can't reproduce the error condition.

In the added test case:

• There is a translatable literal with non-ASCII characters (in a template file)
• The intermediate POT file is created (and preserved for examination)
• When the POT file is created, the header "Content-Type: text/plain; charset=?????\n" is verified and it already has the UFT-8 value for the charset.

Am I missing something? How it comes the created POT file has a "Content-Type: text/plain; charset=CHARSET\n" header?

• Is the fact tha OP is running on Windows?
• Does this happen when extracting literal from .py files? Javascript?

[Ramiro Morales]

Test case

[Claude Paroz]

I think you could just unit test the write_pot_file method with some content like:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE\'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2018-06-07 17:21+0200\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: 8bit\n"

#: django/contrib/gis/apps.py:8
msgid "GIS"
msgstr ""

[Bartosz Grabski]

Thanks Claude, that was actually my idea. Will do.
Ramiro: thanks for the tip.

[Bartosz Grabski]

Updated PR with test: ​https://github.com/django/django/pull/9997

[Tim Graham <timograham@…>]

In 2bc01475:

Fixed #29452 -- Fixed makemessages setting charset of .pot files.

[Claude Paroz]

As this was a regression (read comment:6), I'd be willing to backport this to the 2.1 branch. Any opposition?

[Tim Graham <timograham@…>]

In c7d59825:

[2.1.x] Fixed #29452 -- Fixed makemessages setting charset of .pot files.

Backport of 2bc014750adb093131f77e4c20bc17ba64b75cac from master