UnicodeDecode error when running `makemessages` (python 3.4.1)

[Vidir Valberg Gudmundsson]

Made this model for testing:

from django.db import models
from django.utils.translation import ugettext_lazy as _


class Foo(models.Model):
    bar = models.CharField(
        verbose_name=_('bar'),
        max_length=255
    )

django-admin makemessages -l da creates the following django.po file:

$ cat locale/da/LC_MESSAGES/django.po
# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2014-06-22 19:49+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

#: models.py:7
msgid "bar"
msgstr ""

Changing the last line (doing the translation of "bar") to this:

msgstr "æøå"

Results in this error when running django-admin makemessages -l da again:

processing locale da
Traceback (most recent call last):
  File "./../manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/base.py", line 532, in handle
    return self.handle_noargs(**options)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 295, in handle_noargs
    self.write_po_file(potfile, locale)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 397, in write_po_file
    msgs, errors, status = popen_wrapper(args)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/utils.py", line 25, in popen_wrapper
    output, errors = p.communicate()
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 959, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 1658, in _communicate
    self.stdout.encoding)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 887, in _translate_newlines
    data = data.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 663: ordinal not in range(128)

I'm not sure whether this is a django error or a python one.

[Claude Paroz]

Did you try with other versions of Django?

[Vidir Valberg Gudmundsson]

Just did, and it also happens with 1.6.5 and 1.5.8.

[Vidir Valberg Gudmundsson]

It looks as if my locale was wrong somehow:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Setting LANG to "da_DK.UTF-8" does the trick as far as I can tell.

$ export LANG="da_DK.UTF-8"
$ locale
LANG="da_DK.UTF-8"
LC_COLLATE="da_DK.UTF-8"
LC_CTYPE="da_DK.UTF-8"
LC_MESSAGES="da_DK.UTF-8"
LC_MONETARY="da_DK.UTF-8"
LC_NUMERIC="da_DK.UTF-8"
LC_TIME="da_DK.UTF-8"
LC_ALL="da_DK.UTF-8"
$ django-admin makemessages --traceback -l da
processing language da

[Vidir Valberg Gudmundsson]

PR: ​https://github.com/django/django/pull/2841

[Claude Paroz]

After the night :-), here's how I would have written it: ​https://github.com/django/django/pull/2842

[Tim Graham]

It's not really a Django specific issue, but I guess it wouldn't hurt to mention as we already have a topic guide on unicode.

[Claude Paroz <claude@…>]

In 460ec09d2ed26f5862aabeaa480bcb713d5f331f:

Fixed #22880 -- Added FAQ entry about UnicodeDecodeError

Thanks Víðir Valberg Guðmundsson for the report and
Tim Graham for the review.

[Claude Paroz <claude@…>]

In 198ce3073c763ab17cb5e8d012d64d8ea756282d:

[1.7.x] Fixed #22880 -- Added FAQ entry about UnicodeDecodeError

Thanks Víðir Valberg Guðmundsson for the report and
Tim Graham for the review.
Backport of 460ec09d2e from master.