compilemessages fails

[gregoire.astruc+django@…]

System: WinXP, django 1.5.2 (through virtual-env), python 2.7.3, using Aptana 3.4.2 shell.

When somebody run the following sequence:

$ ./manage.py makemessages -l en
$ ./manage.py compilemessages

compilemessages fails with the following output:

processing file django.po in d:\Trunk\src\locale\en\LC_MESSAGES
Traceback (most recent call last):
  File "d:\v-env\lib\site-packages\django\core\management\base.py", line 222, in run_from_a
    self.execute(*args, **options.__dict__)
  File "d:\v-env\lib\site-packages\django\core\management\base.py", line 255, in execute
    output = self.handle(*args, **options)
  File "d:\v-env\lib\site-packages\django\core\management\commands\compilemessages.py", lin
    compile_messages(self.stderr, locale=locale)
  File "d:\v-env\lib\site-packages\django\core\management\commands\compilemessages.py", lin
    if f.endswith('.po'):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 1: ordinal not in range(128)

Running ./manage.py compilemessages -l en solves the issue and ./manage.py compilemessages appears to work afterwards.

[Claude Paroz]

I've been able to reproduce a similar error in one of my projects.
It might be related to settings.LOCALE_PATH containing non-unicode paths, as os.walk returns unicode results only if it has been fed with an unicode parameter.

Could you test this patch:

--- a/django/core/management/commands/compilemessages.py
+++ b/django/core/management/commands/compilemessages.py
@@ -6,7 +6,7 @@ from optparse import make_option
 
 from django.core.management.base import BaseCommand, CommandError
 from django.core.management.utils import find_command, popen_wrapper
-from django.utils._os import npath
+from django.utils._os import npath, upath
 
 def has_bom(fn):
     with open(fn, 'rb') as f:
@@ -23,7 +23,7 @@ def compile_messages(stdout, locale=None):
     basedirs = [os.path.join('conf', 'locale'), 'locale']
     if os.environ.get('DJANGO_SETTINGS_MODULE'):
         from django.conf import settings
-        basedirs.extend(settings.LOCALE_PATHS)
+        basedirs.extend([upath(path) for path in settings.LOCALE_PATHS])
 
     # Gather existing directories.
     basedirs = set(map(os.path.abspath, filter(os.path.isdir, basedirs)))

[gregoire.astruc+django@…]

At least it compiles.

But there's a little quirk: it now tries to find every single .po on my whole D: drive, which is quite annoying :)

This :

# Gather existing directories.
basedirs = set(map(os.path.abspath, filter(os.path.isdir, basedirs)))
print basedirs

shows:
set([u'd:\\myproject\\myapp\\src\\locale', u'd:\\'])

Hopefully it helps.

[Claude Paroz]

What's in your settings.LOCALE_PATHS?

[gregoire.astruc+django@…]

LOCALE_PATHS = (
    os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', 'locale'))
)

My settings is actually a python module, hence the ../.., so it translates to <project_root>/locale.

[Claude Paroz]

LOCALE_PATHS has to be a list/tuple. Can you test with that fixed?

[furins]

Replying to claudep:

LOCALE_PATHS has to be a list/tuple. Can you test with that fixed?

I've tried to replicate the problem (on a Mac OS 10.7, Django 1.5.5, Python 2.7.5 in a virtualenv).

When settings.LOCALE_PATHS is not a tuple (missing the trailing comma), then compilemessages raises another error, but unfortunately it is a little confusing:

example (with LOCALE_PATHS set erroneously to a string instead of a tuple):

$ python2.7 ./manage.py compilemessages --settings=hotspot.settings
processing file django.po in /Users/stefano/sviluppo/siti/hotspot/locale/it_IT/LC_MESSAGES
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 23: ordinal not in range(128)

as you can see, there are no unicode chars in my path (second line) and also in my .po file (I can assure you). Strange enough, the correct path of my .po file has been identified (/Users/stefano/sviluppo/siti/hotspot/locale/it_IT/LC_MESSAGES) thus at a first glance it seems that compilemessages had read the correct path from the settings. Despite that, the only way to solve this error is to correct settings.LOCALE_PATHS, making it a tuple.

If settings.LOCALE_PATHS is a tuple, a list, or a function returning a similar object - and at least one of the path strings contains unicode chars - then I receive the same error of Gregoire, and I can solve it using Claude's patch.

At the very end: it seems to me that Claude's patch solves the issue. Having LOCALE_PATHS set erroneously to a str object generate another type of error.

[Claude Paroz <claude@…>]

In dffcc5e97988d92b2b8a3bba23a49bcb3cf5d040:

Fixed #20990 -- Ensured unicode paths in compilemessages

Thanks Gregoire Astruc for the report and furins for the review.