Meet my little friends, ANTLR and StringTemplate - USF Computer Science 652 - Programming Languages

requires ANTLR 3.1

Introduction

The goal of this lab is to get you over any fears you may have regarding the use of language tools to solve problems. While you should spend some time looking at the examples shown in this lab, your primary goal is simply to cut and paste the code into files, run ANTLR, compile the resulting Java code, and run the executable to test the application.

This lab starts with a simple grammar to recognize comma-separated values and then gradually modifies the various pieces to get more and more sophisticated output. Even if you don't understand exactly what is going on, blindly follow the cutting and pasting to make the examples work as shown.

Using ANTLR to recognize comma-separated integers

CSV.g

grammar CSV; // must be stored in a file called CSV.g

file: INT (',' INT)* EOF;

INT	: '0'..'9'+ ;
WS	: (' '|'\n'|'\t'|'\r')+ {skip();} ;

Main.java

import java.io.*;
import org.antlr.runtime.*;

public class Main {
    public static void main(String args[]) throws Exception {
        CSVLexer lex = new CSVLexer(new ANTLRInputStream(System.in));
        CommonTokenStream tokens = new CommonTokenStream(lex);
        CSVParser g = new CSVParser(tokens);
        g.file();
    }
}

$ java org.antlr.Tool CSV.g
ANTLR Parser Generator  Version 3.1b1 (??)  1989-2007
$ javac *.java
$

$ java Main
31, 203, 9, 18,
10302, 127
<CTRL-D>
$

Here is the parse tree as created by ANTLRWorks. It shows how the grammar recognized the input.

$ java Main
3 4
<CTRL-D>
line 1:2 extraneous input '4' expecting EOF
$

$ java Main
29, ,
<CTRL-D>
line 1:4 missing INT at ','
$

Returning a list of integers

Modify the start rule to have a return value and then add actions to add integer values to the return list.

file returns [List values=new ArrayList()]
	: a=INT {$values.add($a.int);} // $a.int is v3.1 syntax
	  (',' b=INT {$values.add($b.int);})*
	  EOF
	;

Modify the main to get the list and print it out.

import java.io.*;
import java.util.*;
import org.antlr.runtime.*;

public class Main {
    public static void main(String args[]) throws Exception {
        CSVLexer lex = new CSVLexer(new ANTLRInputStream(System.in));
        CommonTokenStream tokens = new CommonTokenStream(lex);
        CSVParser g = new CSVParser(tokens);
        List values = g.file();
        System.out.println(values);
    }
}

Here is what the output looks like

$ java Main
1, 2, 5, 321
<CTRL-D>
[1, 2, 5, 321]

Generating source code using StringTemplate

Now that you have a list of integers, let's use StringTemplate to emit some nice structured text: an array declaration. Here is the main program that does it. Please note the inline template definition.

import java.io.*;
import java.util.*;
import org.antlr.runtime.*;
import org.antlr.stringtemplate.*;

public class Main {
    public static void main(String args[]) throws Exception {
        CSVLexer lex = new CSVLexer(new ANTLRInputStream(System.in));
        CommonTokenStream tokens = new CommonTokenStream(lex);
        CSVParser g = new CSVParser(tokens);
        List values = g.file();

        String decl =
            "int[] values = { $values; separator=\", \"$ };";
        StringTemplate st = new StringTemplate(decl);
        st.setAttribute("values", values);
        System.out.println(st);
    }
}

Running this, emits:

$ java Main
1, 2, 43
<CTRL-D>
int[] values = { 1, 2, 43 };
$

But, with a big list you get everything on one line:

$ java Main
29, 234, 2, 9, 7, 100, 97, 34, 82, 20, 39, 24, 86, 562,
765, 577, 31, 388, 92, 845, 5707, 422, 82, 20, 39, 24,
86, 562, 765, 577, 5738, 3183, 299, 10002, 2, 9, 7,
100, 97, 34, 82, 20, 39, 24
<CTRL-D>
int[] values = { 29, 234, 2, 9, 7, 100, 97, 34, 82, 20, 39, 24, 86, 562, ... }; // all on one line
$

Modify the template to wrap and starting from the values attribute reference's column.

String decl =
    "int[] values = { $values; wrap, anchor, separator=\", \"$ };";

then tell it to wrap at 72 columns:

System.out.println(st.toString(72));

Now, you get nicely wrapped output:

$ java Main
29, 234, 2, 9, 7, 100, 97, 34, 82, 20, 39, 24, 86, 562,
765, 577, 31, 388, 92, 845, 5707, 422, 82, 20, 39, 24,
86, 562, 765, 577, 5738, 3183, 299, 10002, 2, 9, 7,
100, 97, 34, 82, 20, 39, 24
<CTRL-D>
int[] values = { 29, 234, 2, 9, 7, 100, 97, 34, 82, 20, 39, 24, 86, 562,
                 765, 577, 31, 388, 92, 845, 5707, 422, 82, 20, 39, 24,
                 86, 562, 765, 577, 5738, 3183, 299, 10002, 2, 9, 7, 100,
                 97, 34, 82, 20, 39, 24 };
$