Junior developer

Generating Parsers with JavaCC, Second Edition now available

We're now shipping the second edition of my JavaCC book, Generating Parsers with JavaCC!

Creating this second edition was a labor of love. I went through every page of the first edition and found all the places where JavaCC had moved out from under the book. I've brought those bits up to date, added new sections all over, fixed numerous typos, added more index entries, and generally given it a thorough house cleaning. This second edition also benefits from the intervening two years during which I worked on the JavaCC code and answered questions from folks on the javacc-users mailing list. And I've generally had a little more time to digest some of the information in the first edition, so I think the second edition is improved because of that.

So, what does the second edition include? Here's a sampling:

• Discussion of new JavaCC features like TRACK_TOKENS, TOKEN_EXTENDS, SUPPORT_CLASS_VISIBILITY_PUBLIC, TOKEN_FACTORY, and more.
• Details of new JJTree features like NODE_CLASS, VISITOR_DATA_TYPE, VISITOR_RETURN_TYPE, and more.
• Updated examples to reflect JavaCC's current defaults - e.g., JDK_VERSION is now 1.5 by default, the options block is now optional if empty, and more.
• Updated chapter 8 (Case Study: The JavaCC Grammar) to reflect latest JavaCC grammar.
• Update software versions used throughout - the second edition uses much more recent versions of Java, Maven, JUnit, Jaxen.
• Changes since JavaCC 4.0 are flagged in the index for easy lookup.
• Many small fixes to the text, to examples, to code samples.

I'm excited to get this second edition out there; I think folks will find that it's a nice refresh of the original content. Enjoy!

Lovely SVG railroad diagrams revisited

Last week I blogged about Julian Hyde's Clapham utility which generates nice SVG graphs from JavaCC grammars. One painful bit was that JavaCC didn't output a clean BNF format of a grammar, so you had to do some copying and pasting and general munging to get a grammar in shape for Clapham to use it.

Well, no more. I've added a BNF option to JJDoc (here's an updated javacc.jar) so that it outputs plain old BNF. So now you can easily go from a JavaCC grammar to a Clapham-generated graph:

$ /Users/tom/java.net/javacc/bin/jjdoc -bnf \
-output_file=java.bnf \
/Users/tom/java.net/javacc/examples/JavaGrammars/1.5/Java1.5.jj
Java Compiler Compiler Version 4.3 (Documentation Generator Version 0.1.4)
(type "jjdoc" with no arguments for help)
Reading from file /Users/tom/java.net/javacc/examples/JavaGrammars/1.5/Java1.5.jj . . .
Grammar documentation generated successfully in java.bnf
$ $ java -cp lib/batik-awt-util.jar:lib/batik-bridge.jar:lib/batik-css.jar\
:lib/batik-dom.jar:lib/batik-ext.jar:lib/batik-gvt.jar:lib/batik-parser.jar\
:lib/batik-script.jar:lib/batik-svg-dom.jar:lib/batik-svggen.jar\
:lib/batik-transcoder.jar:lib/batik-util.jar:lib/batik-xml.jar\
:lib/clapham.jar:lib/javacc.jar:lib/junit.jar:lib/xercesImpl.jar net.hydromatic.clapham.Clapham \
-d grammar java.bnf 
Created output directory grammar
Symbol AdditiveExpression 
[... etc ...]

Here's the svg version of one of the nonterminals in the output (using an embed tag):

And the png version:

Fun stuff!

Lovely SVG railroad diagrams with JavaCC and Clapham

Julian Hyde posted to the javacc-users mailing list a few weeks ago about his utility Clapham which generates railroad diagrams. A picture is worth 1K words, so here's the Java grammar rendered using Clapham. Pretty nifty!

The only problem I had was some difficulty in building the utility - so to shortcut that for anyone else, here's a clapham.jar that I built from source. To generate the Java grammar digrams I downloaded Clapham, built that jar file (which you'll want to put in lib/), munged JJDoc's output to create java.bnf and then did this:

java -cp lib/batik-awt-util.jar:lib/batik-bridge.jar:lib/batik-css.jar:lib/batik-dom.jar:lib/batik-ext.jar:lib/batik-gvt.jar:lib/batik-parser.jar:lib/batik-script.jar:lib/batik-svg-dom.jar:lib/batik-svggen.jar:lib/batik-transcoder.jar:lib/batik-util.jar:lib/batik-xml.jar:lib/clapham.jar:lib/javacc.jar:lib/junit.jar:lib/xercesImpl.jar net.hydromatic.clapham.Clapham -d grammar java.bnf

Voila, nice diagrams. Good one, Julian!

This process would be a little easier without the intervening jjdoc step... would be nice to be able to parse a grammar file directly, which would probably mean skipping all the token definitions. Either that or we should write a jjdoc output format that just outputs plain old BNF.

As always, for more info on JavaCC, check out my JavaCC book!

The Eclipse JavaCC plugin and Mac OS X

Lately I've been trying to get the Eclipse JavaCC plugin to work on my Mac 10.5.6 laptop with Eclipse 3.4.2. I was having all sorts of problems... basically the ones which folks are discussing here. After fiddling around a bit I ended up checking out the plugin and rebuilding the plugin.jar file using Java 1.5 and that did the trick. So, to get the Eclipse JavaCC plugin v1.5.13 working, try this:

• Install the JavaCC plugin using the instructions outlined here.
• Download this new plugin.jar file.
• Shut down Eclipse and copy the plugin.jar file into the plugin directory. On my Mac I've got Eclipse installed in /Users/tom/eclipse, so I copied the jar file to /Users/tom/eclipse/plugins/sf.eclipse.javacc_1.5.13/plugin.jar
• Restart Eclipse and create a new grammar file (File => New => Java Project, File => New => File, name it foo.jj)
• Paste in some boilerplace JavaCC grammar contents, e.g.:

options {
  JDK_VERSION="1.5";
}
PARSER_BEGIN(Foo)
public class Foo {}
PARSER_END(Foo)
TOKEN : {
  <HI : "hello">
}
void Foo() : {} {
  <HI>
}

• The file icon should now look like a little "JJ" and the outline view should contain the token definitions and whatnot.

Hopefully that helps someone. And if you're working with JavaCC, check out my JavaCC book!

developer.com Java tool of the year award

Just a quick post to note that JavaCC was apparently in the running for the Developer.com Product of the Year awards for 2009... it's in the "Java Tool" section along with Hudson, Glassfish, and Eclipse SQL Explorer. In the end, it looks like Netbeans won in that category... Tor for the win!

A small fix for JavaCC 4.2's code generation

JavaCC 4.2 was released a few weeks ago, but there was a small problem (spotted by Xavier Le Vourch and patch submitted by rafiyr) which results in the generated JJTree code not using Java 1.5 language constructs.

rafiyr's patch looks good to me, the tests all pass, and a rebuilt javacc.jar file is here if anyone wants to give it a try. I'll commit this in a couple days unless any problems crop up...

JavaCC 4.1 released!

Well, I'm about two months late in blogging about this, but, still: JavaCC 4.1 is out! It's been close to three years since the last release (JavaCC 4.0 came out in Jan 2006), and this new version has a bunch of bug fixes and code cleanups.

The lion's share of the work for this release (and over the past few years in general) was done by Paul Cager, with contributions in the form of patches, discussions, and bug reports from many others. You can download this release here, and feedback is always welcome on the mailing lists.

Finally, I must plug my JavaCC book. It's a perfect Christmas present for your friends who enjoy reading EBNF!

A small JJDoc bug fixed

Michael Iber reported a bug in JavaCC - the documentation generator, JJDoc, was erroring out on tokens defined with repetition ranges. For example, a token definition that contained <FOO>{2,3} would result in this error:

Java Compiler Compiler Version 4.1d1 (Documentation Generator Version 0.1.4)
(type "jjdoc" with no arguments for help)
Reading from file error.jj . . .
Oops: Unknown regular expression type.
Exception in thread "main" java.lang.NullPointerException
        at org.javacc.jjdoc.HTMLGenerator.print(Unknown Source)
        at org.javacc.jjdoc.HTMLGenerator.println(Unknown Source)
        at org.javacc.jjdoc.HTMLGenerator.nonterminalsStart(Unknown Source)
        at org.javacc.jjdoc.JJDoc.emitNormalProductions(Unknown Source)
        at org.javacc.jjdoc.JJDoc.start(Unknown Source)
        at org.javacc.jjdoc.JJDocMain.mainProgram(Unknown Source)
        at org.javacc.jjdoc.JJDocMain.main(Unknown Source)
        at jjdoc.main(Unknown Source)

This is fixed in CVS now (v1.16 of JJDoc.java) and I've posted an updated javacc.jar for those who encounter this problem.

Kirk Pepperdine's nice review of Generating Parsers with JavaCC

I just noticed that Kirk Pepperdine had posted a kind note about my JavaCC book. He talks about having to do write new grammars rarely enough that it's helpful to have a book handy to fire up the neurons again when the task comes up, and I think that's probably a pretty common case. Sounds like the book has been useful to him there.

One thing he says is that Generating Parsers with JavaCC "...has been an invaluable source of information when I've been trying to sort out some of the more obscure error messages that I'm prone to generate." Yup, as I wrote all the grammars for the book I captured lots of error messages as they arose, looked them up in the JavaCC source code, and then added an explanation for each one. I also added those error messages to the index so that you can flip to the back of the book and look up "Multiply defined lexical tokens" or whatever.

Kirk's blog is pretty well known, I think, but I hadn't gone there for a while so it was interesting to catch up with some of his recent posts. I especially enjoyed the one on concurrent mark and sweep garbage collection. I notice in that post's comments that Greg Sporar points over to GCHisto, which has a handrolled parser for GC log files. I wonder if a JavaCC-generated parser would be much faster? Anyhow, good stuff!

Problem trends with PMDReports

Ben Northrup emailed me about his PMDReports project.  To quote from his project page, "Whereas PMD generates and displays code quality statistics at a source code level, PMDReports persists and aggregates these statistics so that code quality can be viewed from a more macro, component level."

I got the latest version (0.8.2) and ran it on PMD's util package; here's the resulting "Component Quality Snapshot Today" report.  You can drill down into the rule violations, and although I just ran it on one component I could see how this could be the makings of a nice dashboard.  It would also be interesting to see those trend graphs after a few days.

Ben's past postings are worth a read; I enjoyed Why Code Quality Tools Work and Why our Programming Gods are so Unkempt?.  Especially the latter one!