JS comments put after statements break make-messages.py output

[Robby Dermody]

To test, make a JS file (say myfile.js) with the following valid JS content:

var a = 1;
if(a != 2 && a != 5) //this comment breaks the file
{
    //this does not
    alert(gettext("foobar"));
}

Running make-messages.py -d djangojs -a will then yield the following output for that (in the myfile.js.py intermediate file it produces):

var a = 1;
if(a != 2 && a != 5) //this comment breaks the file
{
#this does not
    alert(gettext("foobar"));
}

As you can see, the comment after the if statement was not replaced, and since xgettext is then run in Perl mode, it seems to choke on that input. The result depends on the exact code: This example will cause only that next gettext("foobar") not to be generated (ones further down in the code will). With other code I had that had a similar line, nothing was generated. The failure is silent and the only way to know is by checking the gettext output (or lack thereof :).

This is due to the regexp in make-messages: pythonize_re = re.compile(r'\n\s*//') and then the replacement code: src = pythonize_re.sub('\n#', src)

That assumes that comments come after newlines. I'm not submitting a patch right now because I'm unsure about the best regexp to use for this that will get all the valid JS comment cases (or if that is even something the django devs want to do). At the very least, if you all choose not to address this in the code, there should be a note in the documentation telling folks to always put JS comments on their own lines.

As make-messages.py is now included in django-admin AFAIK, I've categorized it to that.

[anonymous]

I also faced this *bug*.

It also seems that class comments like this one :

/**
 * ***************************** 
 * AddModule main / window
 * @constructor
 * @class MyDesktop.AddModule
 * *****************************
 */ 

breaks the following gettext references. If I remove the "/" in " * AddModule main / window" .. it works.

[Sung-jin Hong]

Seems to me like a correct and reproducible bug.

[anonymous]

This feature is completely broken at the current state.

[anonymous]

I just quite some time on trying to find out why some strings from my JS files weren't translated.
I was aware of the end of line comment probleme and some other as I post the 07/11/08 08:52:41 comment.

Any way something else was breaking the process.
I figured out that this line

,this.split = elem.split ? true : false

among others causes problem.

Anyway. I asked myself why only "comment out" the JS comments before processing the fake "js.py" file with gettext ?
Seems to me that we can "comment out" all the lines that don't have gettext or ngettext in it.

pythonize_re2 = re.compile('gettext')
src = open(os.path.join(dirpath, file), "r").readlines()
dest = open(os.path.join(dirpath, '%s.py' % file), "wb")
for line in src :
 if not pythonize_re2.search(line):
  dest.write('#%s' % line)
 else:
  dest.write(line)
dest.close()

I went line by line due to my regex bad competences. a bit slower.
I made a quick

grep -r 'gettext' /js/templates | grep '?'

then I change some lines

return values.value ? gettext("Oui") : gettext("Non");

to 

return values.value ? 
 gettext("Oui") : gettext("Non");  

I looked at the latest translation of all JS files to see if 1- it was present (not the case before) or 2- line number was correct.
All files were well parse

Not a solution a quick hack.

xav

[Ned Batchelder]

For the curious, the bad code blocks in the original description say:

To test, make a JS file (say myfile.js) with the following valid JS content:

    var a = 1;
    if(a != 2 && a != 5) //this comment breaks the file
    {
	//this does not
	alert(gettext("foobar"));
    }

Running make-messages.py -d djangojs -a will then yield the following output for that (in the myfile.js.py intermediate file it produces):

    var a = 1;
    if(a != 2 && a != 5) //this comment breaks the file
    {
    #this does not
	alert(gettext("foobar"));
    }

[Łukasz Rekucki]

Fixed formating in description :)

[Ned Batchelder]

Adapt makemessages to use JsLexer

[Ned Batchelder]

Two patches attached: jslex.diff adds a Javascript lexer, with tests, and makemessages.diff uses the new lexer to process Javascript files. This also fixes #14045, #15495, and #15331.

[Jannis Leidel]

As mentioned on the developers mailing list, I strongly believe that refactoring the i18n tools to use Babel for message extraction instead of shipping an own JavaScript lexer is the favorable way.

[Ned Batchelder]

In the absence of someone working to get Babel integrated with Django, rejecting this patch is the perfect being the enemy of the good, no? Can you identify a problem with this patch? There are lots of problems with the existing trunk code.

[Jannis Leidel]

Replying to nedbatchelder:

Can you identify a problem with this patch?

Yes, we'd introduce a huge chunk of code that would further manifest the xgettext hack. In other words, I'm not convinced that switching the hack from the Perl to C lexer in gettext is the right approach to solve this problem.

[Ned Batchelder]

I understand the philosophical concern. I'm wondering if there's any observable incorrect behavior in the code.

[Marc Demierre <marc.demierre@…>]

As the actual documentation does not even state that there are limitations with the message extraction from javascript files, I think that this patch should be used at least until the transition to Babel. The expected behaviour is to extract all messages wrapped in the gettext() function.

The only real solution to the current problem for a developer is not to use the makemessages utility for javascript at all. I think that using the patch would be better than that.

If the patch is not accepted, I propose to update the documentation to clearly state that javacript parsing is a hack and that it is not working properly.

[Ned Batchelder]

Updated patch: deals with unicode escapes in ids, and fix a doctest.

[Jannis Leidel]

In [16333]:

Fixed #7704, #14045 and #15495 -- Introduce a lexer for Javascript to fix multiple problems of the translation of Javascript files with xgettext. Many thanks to Ned Batchelder for his contribution of the JsLex library.

[Aymeric Augustin]

In [17515]:

Fixed #17451 -- Mentioned the new JavaScript lexer in the release notes. Refs #7704.