Regression in `makessages -d django` when processing multi-line comments with non-ASCII characters

[Ramiro Morales]

As reproted Denis Drescher via email:

The fix committed in [15882] causes problems with comments that contain Unicode characters. The error is:

...
File ".../django/utils/translation/trans_real.py", line 450, in templatize
   content = u''.join(comment)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
131: ordinal not in range(128)

To reproduce it, just add a comment block to a template that contains a string such as this one, where the apostrophe is non-ASCII: “It doesn’t work properly on our page.”

To sort of fix it, I removed the three “u”s in lines 450, 457, and 459 of the file, but I read somewhere that the way to go is to use Unicode consistently everywhere, so this solution may not be ideal.

[Ramiro Morales]

Tests for this bug

[Ramiro Morales]

In [16038]:

Fixed #15848 -- Fixed regression introduced in [15882] in makemessages management command when processing multi-line comments that contain non-ASCCI characters in templates. Thanks for the report Denis Drescher.

[Ramiro Morales]

In [16040]:

[1.3.X] Fixed #15848 -- Fixed regression introduced in [15882] in makemessages management command when processing multi-line comments that contain non-ASCCI characters in templates. Thanks for the report Denis Drescher.

Backport of r16038/r16039 from trunk.

[anonymous]

I'm using offical 1.3 (pip install django) and still get this error. Should I upgrade to trunk from svn?

  File "./manage.py", line 11, in <module>
    execute_manager(settings)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 365, in handle_noargs
    make_messages(locale, domain, verbosity, process_all, extensions, symlinks, ignore_patterns, no_wrap, no_obsolete)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 233, in make_messages
    f.write(templatize(src, orig_file[2:]))
  File "/usr/local/lib/python2.7/dist-packages/django/utils/translation/__init__.py", line 127, in templatize
    return _trans.templatize(src, origin)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/translation/trans_real.py", line 450, in templatize
    content = u''.join(comment)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)

[Ramiro Morales]

Replying to anonymous:

I'm using offical 1.3 (pip install django) and still get this error. Should I upgrade to trunk from svn?

As you can see from the two commit messages before your question, the issue is fixed both in trunk and the releases/1.3.X branch. You can use any of them although I'd suggest the latter because it receives only this kind of bug fixes.