UnicodeDecodeError with makemessages

[Dirk Eschler]

Running makemessages in the project folder throws a UnicodeDecodeError here with django-1.3. Everything works fine with django-1.2.

$ ./manage.py makemessages -l de
processing language de
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_manager(settings)
  File "/path/to/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/path/to/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/path/to/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/path/to/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/path/to/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/path/to/django/core/management/commands/makemessages.py", line 365, in handle_noargs
    make_messages(locale, domain, verbosity, process_all, extensions, symlinks, ignore_patterns, no_wrap, no_obsolete)
  File "/path/to/django/core/management/commands/makemessages.py", line 233, in make_messages
    f.write(templatize(src, orig_file[2:]))
  File "/path/to/django/utils/translation/__init__.py", line 127, in templatize
    return _trans.templatize(src, origin)
  File "/path/to/django/utils/translation/trans_real.py", line 450, in templatize
    content = u''.join(comment)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 284: ordinal not in range(128)

Is this in any way related to the changes made in project level translation handling mentioned in the 1.3 release notes? I'm not sure if i understand them correctly, does it mean that project level should be avoided in favour of app level translations in general? Adding the project locale folder to LOCALE_PATHS doesn't make a difference by the way, same UnicodeDecodeError.

[James Aylett]

I've had this too; it should (?) only happen with non-ASCII characters in templates. The problem is that higher up the call stack (makemessages.py:233) src is a str (it's read straight out of a file), so that:

    content = u''.join(comment)

doesn't work, because the members of comment are actually str not unicode. Changing django/utils/translation/trans_real.py:450 to the following fixes it for me:

    content = u''.join([l.decode('utf-8') for l in comment])

An actual fix is going to be a bit more subtle than that…

[anonymous]

Just ran into the same problem. I relocated the locale folder to my main application instead of my project root but still got the same error.

I managed to pin point the offending code by removing all the templates from the template folder and then put them back one by one while doing a makemessage -a between each of them.

It turned out that the error was triggered by a single file, which didn't seem to contain non-ascii character in any trans block.. So I did the same for the code, I removed it all and started to put it back chunk by chunk while doing a makemessages -a between each chunks.

Which led me to find out that the offending non-ascii character was in a TEMPLATE COMMENT.

IMO the translation module should ignore anything that isn't in a trans/blocktrans altogether. In my case we are French, so often in template we've put French comments. I'll probably have a lots of projects that will start to break the translation module because of this this..

[Jonas H.]

Duplicate of #15848