ANTLR, heterogeneous AST problem

Question

I examine heterogeneous trees in ANTLR (using ANTLRWorks 1.4.2).

Here is the example of what I have already done in ANTLR.

grammar test;

options {
    language = java;
    output = AST;
}

tokens {
    PROGRAM;
    VAR;
}

@members {
    class Program extends CommonTree {
        public Program(int ttype) {
            token = new CommonToken(ttype, "<start>");
        }
    }
}

start
    :    program var function
        // Works fine:
        //->    ^(PROGRAM program var function)

        // Does not work (described below):
        ->    ^(PROGRAM<Program> program var function)
    ;

program
    :    'program'! ID ';'!
    ;

var
    :    TYPE^ ID ';'!
    ;

function
    :    ID '('! ')'! ';'!
    ;

TYPE
    :    'int'
    |    'string'
    ;

ID
    :    ('a'..'z' | 'A'..'Z')+
    ;

WHITESPACE
    :    (' ' | '\t' '\n'| '\r' | '\f')+ {$channel = HIDDEN;}
    ;

Sample input:

program foobar;
int foo;
bar();

When I use rewrite rule ^(PROGRAM<Program> program var function), ANTLR stumbles over and I get AST like this:

AntLR

Whereas when I use this rewrite rule ^(PROGRAM program var function) it works:

enter image description here

Could anyone explain where am I wrong, please? Frankly, I do not really get the idea of heterogeneous trees and how do I use <…> syntax in ANTLR.
What do r0 and r1 mean (first picture)?

AFAIK, you can't just mix < ... > inside rewrite rules: you have either output=AST; or output=template; defined in the options{...} section, not both. So you either generate an AST, or are generating (source) code with the StringTemplate engine. What exactly are you tying to do here anyway? What is your goal? — Bart Kiers
@Bart Thanks for the comment & edit! Based on example here, I can mix <…> and rewrite rules. My goal is to build heterogeneous AST (as the part of assignment). As far as I understand (it is tricky!), I have to implement my own tree and AST nodes. No Java code is necessary at this point (except for the code in the @members section). Spent two days, no luck so far. — Little Jeans
Aha, this is new v3.1+ syntax: my ANTLR book (compliant with v3.0) only mentions < ... > in combination with output=template. Thanks for the link. I don't have time at the moment to go through it, but I will later on (tomorrow probably) and get back to you. — Bart Kiers
@Bart Thank you. What book do you mean? I have this one — «The Definitive ANTLR Reference» (The Pragmatic Programmers), although did not find any useful information. — Little Jeans
Yeah sorry, "the ANTLR book"is indeed The Definitive ANTLR Reference from Prag-Prog. In there, < ... > is only used in chapter 9, Generating Structured Text with Templates and Grammars, where you need to set output=template in the options section. This new feature isn't handled, AFAIK. — Bart Kiers

Bart Kiers Bart Kiers · Accepted Answer · 2011-04-13T14:13:10

I have no idea what these r0 and r1 mean: I don't use ANTLRWorks for debugging, so can't comment on that.

Also, language = java; causes ANTLR 3.2 to produce the error:

error(10): internal error: no such group file java.stg

error(20): cannot find code generation templates java.stg
error(10): internal error: no such group file java.stg

error(20): cannot find code generation templates java.stg

ANTLR 3.2 expects it to be language = Java; (capital "J"). But, by default the target is Java, so, mind as well remove the language = ... entirely.

Now, as to you problem: I cannot reproduce it. As I mentioned, I tested it with ANTLR 3.2, and removed the language = java; part from your grammar, after which everything went as (I) expected.

Enabling the rewrite rule -> ^(PROGRAM<Program> program var function) produces the following ATS:

enter image description here

and when enabling the rewrite rule -> ^(PROGRAM program var function) instead, the following AST is created:

enter image description here

I tested both rewrite rules this with the following class:

import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;

public class Main {
    public static void main(String[] args) throws Exception {
        ANTLRStringStream in = new ANTLRStringStream("program foobar; int foo; bar();");
        testLexer lexer = new testLexer(in);
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        testParser parser = new testParser(tokens);
        testParser.start_return returnValue = parser.start();
        CommonTree tree = (CommonTree)returnValue.getTree();
        DOTTreeGenerator gen = new DOTTreeGenerator();
        StringTemplate st = gen.toDOT(tree);
        System.out.println(st);
    }
}

And the images are produced using graph.gafol.net (and the output of the Main class, of course).

ANTLR, heterogeneous AST problem

1 Answers