django-admin.py makemessages fails on multiline blocktrans tags on Windows

[Mihai Damian]

Running make-messages.py on the Django SVN version on Windows produces the following error:

errors happened while running xgettext on bookmarklets.html
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence

The error is caused by \r\n line terminators in Windows that get passed to xgettext.
As make-messages.py calls django.utils.translation.templatize to produce the input files for xgettext, maybe the os.name could be checked there and have the \r stripped.

[Mihai Damian]

patch for django.utils.translation.templatize

[Mihai Damian]

Added possible patch for the templatize function. Checked it on Windows and it worked.

[Ramiro Morales]

My try at fixing this problem. Tested with the Django catalogs under WIndows and Linux.

[Michael Radziej]

This should work for Windows beta testers, so I set the mile stone to 1.0beta.

[Ramiro Morales]

Had forgot about this one. I've attached a patch updating the approach I propose to trunk post-r7844.

You might also want to apply this minimal additional modification to the same file (didn't want to include this in the patch because it's strictly unrelated):

django/core/management/commands/makemessages.py

@@ -68 +68 @@
                     sys.stdout.write('processing file %s in %s\n' % (file, dirpath))
                 src = open(os.path.join(dirpath, file), "rU").read()
                 src = pythonize_re.sub('\n#', src)
-                open(os.path.join(dirpath, '%s.py' % file), "wt").write(src)
                 thefile = '%s.py' % file
+                open(os.path.join(dirpath, thefile), "wt").write(src)
                 cmd = 'xgettext -d %s -L Perl --keyword=gettext_noop --keyword=gettext_lazy --keyword=ngettext_lazy:1,2 --from-code UTF-8 -o - "%s"' % (domain, os.path.join(dirpath, thefile))
                 (stdin, stdout, stderr) = os.popen3(cmd, 't')
                 msgs = stdout.read()

[Malcolm Tredinnick]

There's not documented "t" option to open(), so what are you trying to do there?

[Ramiro Morales]

Replying to mtredinnick:

There's not documented "t" option to open(), so what are you trying to do there?

My bad, I don't know where I the got the 't's, what I was trying to do was just eliminate 'b'. I'm attaching a corrected version of the patch updated to r8255. It doesn't break things under Linux and corrects multi-line msgid extraction under Windows, in both cases generating the full set of Django django.po catalogs.

[Malcolm Tredinnick]

Ramiro: now I have another question... Aren't those write changes going to affect things as the files move between Windows and linux, say? Since you're now writing out with native line-endings, rather than the same line ending in every case (as I understand the change), won't this mean that a PO file generated on Linux and then updated on Windows will come back with a patch that changes every line ending? That would be bad.

It's possible I'm not understanding the effect there, but we do have a number of locale files that are translated by people using Windows, Linux and Mac systems and I don't want a run of patches that only mess around with line endings (which I thought we just fixed last year). You're the guy with the experience here, so I'll trust your conclusions, but can you confirm you haven't made things less portable in this fashion.

[Malcolm Tredinnick]

Moving to the 1.0 milestone. We should definitely fix this as soon as we can; it's a real bug. But it's not something that will disrupt things so much that it absolutely must be in beta.

[Ramiro Morales]

New version of patch, makemessages always creates/update .po files with Unix line-endings, even on Windows

[Ramiro Morales]

Replying to mtredinnick:

Ramiro: now I have another question... Aren't those write changes going to affect things as the files move between Windows and linux, say? Since you're now writing out with native line-endings, rather than the same line ending in every case (as I understand the change), won't this mean that a PO file generated on Linux and then updated on Windows will come back with a patch that changes every line ending? That would be bad.

That is exactly what was happening, thanks for noting it. I've modified things even further so makemessages command always creates the .po catalogs with Unix file-endings. See attached patch.

I´ve tested it under Linux and Windows both creating django and djangojs .po files from scratch and updating existing ones.

While loking at this I found the script doesn´t cleanup after istelf when it finds an error executing the gettext tools. I've opened a ticket for that (#8536).

One minor (not critical at all) note: In a .po file created or updated under Windows, location comments located above every msgid will have the form

#: .\conf\global_settings.py:44
msgid "Arabic"
...

instead of

#: conf/global_settings.py:44
msgid "Arabic"
...

[Malcolm Tredinnick]

Fixed in [8576].

[Jacob]

Milestone 1.0 deleted