More JavaCC optimizations

Paul Cager has been improving JavaCC again - this time he reduced the amount of object allocation done by a JavaCC-generated lexer. 

This began with a nicely detailed bug filed by s_fuhrm that showed that a new StringBuffer is being recreated for every token that's parsed when we could really just reuse one StringBuffer and clear it out after each match.  The change that Paul implemented is especially nice in that it also eliminates an if statement (a null check), so that's an extra performance boost.

The only gotcha is that if you've been using the image variable in your lexical actions, you'll start getting different results.  For example, suppose you had a lexical specification like this:

TOKEN_MGR_DECLS : {
  static StringBuffer lastB = new StringBuffer();
  static void p() {
    System.out.println("lastB is : " + lastB);
  }
}
TOKEN : {
  <A : "a"> { p(); }
  | <B : "b" (["1"-"9"])* > { p() ; lastB = image; }
}

With JavaCC 4.0, the image would never be reused and with input data of b12 a b42 you'd get output like this:

lastB is : 
lastB is : b12
lastB is : b12

In other words, that image object that lastB is referencing would stick around.  With this change in place, image (like the Matrix) is reloaded and you'll get this:

lastB is : 
lastB is : a
lastB is : b42

One solution is to use matchedToken.image instead - or you could just call toString on the image reference to get a copy of the String.    You can see an example of this on page 59 of Generating Parsers with JavaCC.  Finally, if you want to give your grammar a whirl with this change, I've posted a new javacc.jar built from the latest code in CVS here.  Enjoy!

Comments

[Sharath Reddy]

Hi Tom,

I'm a software engineer from Hyderabad, India. Not sure if this is the appropriate place to post this, but wanted to thank you for PMD. We find it a very valuable tool, especially with many junior developers in the team.

I've contributed to open source projects in the past (was a committer to Apache MyFaces) and plan to get involved with PMD too.

[tomcopeland]

Sharath, thanks for the kind words! Glad to hear PMD has been useful for you guys; good times. Xavier Le Vourch is really leading the PMD charge these days, and I'm sure he'd be happy if you can lend a hand with PMD's worklist; there's certainly lots to do there!