Opened 16 years ago

Closed 16 years ago

Last modified 12 years ago

#7084 closed (fixed)

django-admin.py makemessages fails on multiline blocktrans tags on Windows

Reported by: Mihai Damian Owned by: nobody
Component: Internationalization Version: dev
Severity: Keywords: make-messages xgettext new line \r makemessages
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Running make-messages.py on the Django SVN version on Windows produces the following error:

errors happened while running xgettext on bookmarklets.html
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence

The error is caused by \r\n line terminators in Windows that get passed to xgettext.
As make-messages.py calls django.utils.translation.templatize to produce the input files for xgettext, maybe the os.name could be checked there and have the \r stripped.

Attachments (5)

trans_real.py (2.0 KB ) - added by Mihai Damian 16 years ago.
patch for django.utils.translation.templatize
t7084.diff (1.9 KB ) - added by Ramiro Morales 16 years ago.
My try at fixing this problem. Tested with the Django catalogs under WIndows and Linux.
t7084_post_r7844.diff (1.9 KB ) - added by Ramiro Morales 16 years ago.
t7084-r8255.diff (2.0 KB ) - added by Ramiro Morales 16 years ago.
t7084-r8534.diff (2.9 KB ) - added by Ramiro Morales 16 years ago.
New version of patch, makemessages always creates/update .po files with Unix line-endings, even on Windows

Download all attachments as: .zip

Change History (17)

by Mihai Damian, 16 years ago

Attachment: trans_real.py added

patch for django.utils.translation.templatize

comment:1 by Mihai Damian, 16 years ago

Has patch: set
Needs tests: set

Added possible patch for the templatize function. Checked it on Windows and it worked.

by Ramiro Morales, 16 years ago

Attachment: t7084.diff added

My try at fixing this problem. Tested with the Django catalogs under WIndows and Linux.

comment:2 by Marc Fargas, 16 years ago

Triage Stage: UnreviewedAccepted

comment:3 by Michael Radziej, 16 years ago

milestone: 1.0 beta

This should work for Windows beta testers, so I set the mile stone to 1.0beta.

by Ramiro Morales, 16 years ago

Attachment: t7084_post_r7844.diff added

comment:4 by Ramiro Morales, 16 years ago

Had forgot about this one. I've attached a patch updating the approach I propose to trunk post-r7844.

You might also want to apply this minimal additional modification to the same file (didn't want to include this in the patch because it's strictly unrelated):

  • django/core/management/commands/makemessages.py

    diff -r 1b29e4bb845e django/core/management/commands/makemessages.py
    a b  
    6868                    sys.stdout.write('processing file %s in %s\n' % (file, dirpath))
    6969                src = open(os.path.join(dirpath, file), "rU").read()
    7070                src = pythonize_re.sub('\n#', src)
    71                 open(os.path.join(dirpath, '%s.py' % file), "wt").write(src)
    7271                thefile = '%s.py' % file
     72                open(os.path.join(dirpath, thefile), "wt").write(src)
    7373                cmd = 'xgettext -d %s -L Perl --keyword=gettext_noop --keyword=gettext_lazy --keyword=ngettext_lazy:1,2 --from-code UTF-8 -o - "%s"' % (domain, os.path.join(dirpath, thefile))
    7474                (stdin, stdout, stderr) = os.popen3(cmd, 't')
    7575                msgs = stdout.read()

comment:5 by Malcolm Tredinnick, 16 years ago

Needs tests: unset
Patch needs improvement: set

There's not documented "t" option to open(), so what are you trying to do there?

in reply to:  5 comment:6 by Ramiro Morales, 16 years ago

Replying to mtredinnick:

There's not documented "t" option to open(), so what are you trying to do there?

My bad, I don't know where I the got the 't's, what I was trying to do was just eliminate 'b'. I'm attaching a corrected version of the patch updated to r8255. It doesn't break things under Linux and corrects multi-line msgid extraction under Windows, in both cases generating the full set of Django django.po catalogs.

by Ramiro Morales, 16 years ago

Attachment: t7084-r8255.diff added

comment:7 by Malcolm Tredinnick, 16 years ago

Ramiro: now I have another question... Aren't those write changes going to affect things as the files move between Windows and linux, say? Since you're now writing out with native line-endings, rather than the same line ending in every case (as I understand the change), won't this mean that a PO file generated on Linux and then updated on Windows will come back with a patch that changes every line ending? That would be bad.

It's possible I'm not understanding the effect there, but we do have a number of locale files that are translated by people using Windows, Linux and Mac systems and I don't want a run of patches that only mess around with line endings (which I thought we just fixed last year). You're the guy with the experience here, so I'll trust your conclusions, but can you confirm you haven't made things less portable in this fashion.

comment:8 by Malcolm Tredinnick, 16 years ago

milestone: 1.0 beta1.0

Moving to the 1.0 milestone. We should definitely fix this as soon as we can; it's a real bug. But it's not something that will disrupt things so much that it absolutely must be in beta.

by Ramiro Morales, 16 years ago

Attachment: t7084-r8534.diff added

New version of patch, makemessages always creates/update .po files with Unix line-endings, even on Windows

in reply to:  7 comment:9 by Ramiro Morales, 16 years ago

Replying to mtredinnick:

Ramiro: now I have another question... Aren't those write changes going to affect things as the files move between Windows and linux, say? Since you're now writing out with native line-endings, rather than the same line ending in every case (as I understand the change), won't this mean that a PO file generated on Linux and then updated on Windows will come back with a patch that changes every line ending? That would be bad.

That is exactly what was happening, thanks for noting it. I've modified things even further so makemessages command always creates the .po catalogs with Unix file-endings. See attached patch.

I´ve tested it under Linux and Windows both creating django and djangojs .po files from scratch and updating existing ones.

While loking at this I found the script doesn´t cleanup after istelf when it finds an error executing the gettext tools. I've opened a ticket for that (#8536).

One minor (not critical at all) note: In a .po file created or updated under Windows, location comments located above every msgid will have the form

#: .\conf\global_settings.py:44
msgid "Arabic"
...

instead of

#: conf/global_settings.py:44
msgid "Arabic"
...

comment:10 by Ramiro Morales, 16 years ago

Keywords: makemessages added
Summary: make-messages fails on multiline blocktrans tags on Windowsdjango-admin.py makemessages fails on multiline blocktrans tags on Windows

comment:11 by Malcolm Tredinnick, 16 years ago

Resolution: fixed
Status: newclosed

Fixed in [8576].

comment:12 by Jacob, 12 years ago

milestone: 1.0

Milestone 1.0 deleted

Note: See TracTickets for help on using tickets.
Back to Top