Code

Opened 6 years ago

Closed 6 years ago

Last modified 3 years ago

#7084 closed (fixed)

django-admin.py makemessages fails on multiline blocktrans tags on Windows

Reported by: MihaiD Owned by: nobody
Component: Internationalization Version: master
Severity: Keywords: make-messages xgettext new line \r makemessages
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

Running make-messages.py on the Django SVN version on Windows produces the following error:

errors happened while running xgettext on bookmarklets.html
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence
xgettext: internationalized messages should not contain the `\r' escape sequence

The error is caused by \r\n line terminators in Windows that get passed to xgettext.
As make-messages.py calls django.utils.translation.templatize to produce the input files for xgettext, maybe the os.name could be checked there and have the \r stripped.

Attachments (5)

trans_real.py (2.0 KB) - added by MihaiD 6 years ago.
patch for django.utils.translation.templatize
t7084.diff (1.9 KB) - added by ramiro 6 years ago.
My try at fixing this problem. Tested with the Django catalogs under WIndows and Linux.
t7084_post_r7844.diff (1.9 KB) - added by ramiro 6 years ago.
t7084-r8255.diff (2.0 KB) - added by ramiro 6 years ago.
t7084-r8534.diff (2.9 KB) - added by ramiro 6 years ago.
New version of patch, makemessages always creates/update .po files with Unix line-endings, even on Windows

Download all attachments as: .zip

Change History (17)

Changed 6 years ago by MihaiD

patch for django.utils.translation.templatize

comment:1 Changed 6 years ago by MihaiD

  • Has patch set
  • Needs documentation unset
  • Needs tests set
  • Patch needs improvement unset

Added possible patch for the templatize function. Checked it on Windows and it worked.

Changed 6 years ago by ramiro

My try at fixing this problem. Tested with the Django catalogs under WIndows and Linux.

comment:2 Changed 6 years ago by telenieko

  • Triage Stage changed from Unreviewed to Accepted

comment:3 Changed 6 years ago by mir

  • milestone set to 1.0 beta

This should work for Windows beta testers, so I set the mile stone to 1.0beta.

Changed 6 years ago by ramiro

comment:4 Changed 6 years ago by ramiro

Had forgot about this one. I've attached a patch updating the approach I propose to trunk post-r7844.

You might also want to apply this minimal additional modification to the same file (didn't want to include this in the patch because it's strictly unrelated):

diff -r 1b29e4bb845e django/core/management/commands/makemessages.py
--- a/django/core/management/commands/makemessages.py   Tue Jul 22 15:40:25 2008 -0300
+++ b/django/core/management/commands/makemessages.py   Tue Jul 22 15:48:06 2008 -0300
@@ -68,8 +68,8 @@
                     sys.stdout.write('processing file %s in %s\n' % (file, dirpath))
                 src = open(os.path.join(dirpath, file), "rU").read()
                 src = pythonize_re.sub('\n#', src)
-                open(os.path.join(dirpath, '%s.py' % file), "wt").write(src)
                 thefile = '%s.py' % file
+                open(os.path.join(dirpath, thefile), "wt").write(src)
                 cmd = 'xgettext -d %s -L Perl --keyword=gettext_noop --keyword=gettext_lazy --keyword=ngettext_lazy:1,2 --from-code UTF-8 -o - "%s"' % (domain, os.path.join(dirpath, thefile))
                 (stdin, stdout, stderr) = os.popen3(cmd, 't')
                 msgs = stdout.read()

comment:5 follow-up: Changed 6 years ago by mtredinnick

  • Needs tests unset
  • Patch needs improvement set

There's not documented "t" option to open(), so what are you trying to do there?

comment:6 in reply to: ↑ 5 Changed 6 years ago by ramiro

Replying to mtredinnick:

There's not documented "t" option to open(), so what are you trying to do there?

My bad, I don't know where I the got the 't's, what I was trying to do was just eliminate 'b'. I'm attaching a corrected version of the patch updated to r8255. It doesn't break things under Linux and corrects multi-line msgid extraction under Windows, in both cases generating the full set of Django django.po catalogs.

Changed 6 years ago by ramiro

comment:7 follow-up: Changed 6 years ago by mtredinnick

Ramiro: now I have another question... Aren't those write changes going to affect things as the files move between Windows and linux, say? Since you're now writing out with native line-endings, rather than the same line ending in every case (as I understand the change), won't this mean that a PO file generated on Linux and then updated on Windows will come back with a patch that changes every line ending? That would be bad.

It's possible I'm not understanding the effect there, but we do have a number of locale files that are translated by people using Windows, Linux and Mac systems and I don't want a run of patches that only mess around with line endings (which I thought we just fixed last year). You're the guy with the experience here, so I'll trust your conclusions, but can you confirm you haven't made things less portable in this fashion.

comment:8 Changed 6 years ago by mtredinnick

  • milestone changed from 1.0 beta to 1.0

Moving to the 1.0 milestone. We should definitely fix this as soon as we can; it's a real bug. But it's not something that will disrupt things so much that it absolutely must be in beta.

Changed 6 years ago by ramiro

New version of patch, makemessages always creates/update .po files with Unix line-endings, even on Windows

comment:9 in reply to: ↑ 7 Changed 6 years ago by ramiro

Replying to mtredinnick:

Ramiro: now I have another question... Aren't those write changes going to affect things as the files move between Windows and linux, say? Since you're now writing out with native line-endings, rather than the same line ending in every case (as I understand the change), won't this mean that a PO file generated on Linux and then updated on Windows will come back with a patch that changes every line ending? That would be bad.

That is exactly what was happening, thanks for noting it. I've modified things even further so makemessages command always creates the .po catalogs with Unix file-endings. See attached patch.

I´ve tested it under Linux and Windows both creating django and djangojs .po files from scratch and updating existing ones.

While loking at this I found the script doesn´t cleanup after istelf when it finds an error executing the gettext tools. I've opened a ticket for that (#8536).

One minor (not critical at all) note: In a .po file created or updated under Windows, location comments located above every msgid will have the form

#: .\conf\global_settings.py:44
msgid "Arabic"
...

instead of

#: conf/global_settings.py:44
msgid "Arabic"
...

comment:10 Changed 6 years ago by ramiro

  • Keywords makemessages added
  • Summary changed from make-messages fails on multiline blocktrans tags on Windows to django-admin.py makemessages fails on multiline blocktrans tags on Windows

comment:11 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in [8576].

comment:12 Changed 3 years ago by jacob

  • milestone 1.0 deleted

Milestone 1.0 deleted

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.