ANTLR java test file can't create object of tree grammar

Question

I am creating a parser using ANTLR 3.x that targets java. I have written both parser grammar (for creating Abstract Syntax Tree, AST) and Tree Grammar (for performing operations on AST). Finally, to test both grammar files, I have written a test file in Java.

Have a look at the below code,

protocol grammar

grammar protocol;
options {
      language = Java;
  output = AST;
}

tokens{ //imaginary tokens
PROT;
INITIALP;
PROC;
TRANSITIONS;
}
@header {
import twoprocess.Configuration;
package com.javadude.antlr3.x.tutorial;
}

@lexer::header {
  package com.javadude.antlr3.x.tutorial;
}
/*
parser rules, in lowercase letters
*/
program
    : declaration+
    ;
declaration
    :protocol
    |initialprocess
    |process
    |transitions
    ;

protocol
    :'protocol' ID ';' -> ^(PROT ID)
    ;
initialprocess
    :'pin' '=' INT ';' -> ^(INITIALP INT)
    ;
process
    :'p' '=' INT ';' -> ^(PROC INT)
    ;
transitions
    :'transitions' '=' INT ('(' INT ',' INT ')') + ';' -> ^(TRANSITIONS INT INT INT*)
    ;

/*
lexer rules (tokens), in upper case letters
*/
ID  
    : (('a'..'z' | 'A'..'Z'|'_')('a'..'z' | 'A'..'Z'|'0'..'9'|'_'))*;
INT 
    : ('0'..'9')+;
WHITESPACE
    : ('\t' | ' ' | '\r' | '\n' | '\u000C')+ {$channel = HIDDEN;};

protocolWalker

grammar protocolWalker;

options {
  language = Java;
  //Error, eclipse can't access tokenVocab named protocol
  tokenVocab = protocol;    //import tokens from protocol.g i.e, from protocol.tokens file
  ASTLabelType = CommonTree;
  }

@header {
import twoprocess.Configuration;
package com.javadude.antlr3.x.tutorial;
}

program
    : declaration+
    ;

declaration
    :protocol
    |initialprocess
    |process
    |transitions
    ;

protocol
    :^(PROT ID)
    {System.out.println("create protocol " +$ID.text);}
    ;

initialprocess
    :^(INITIALP INT)
    {System.out.println("");}
    ;

process
    :^(PROC INT)
    {System.out.println("");}
    ;

transitions
    :^(TRANSITIONS INT INT INT*)
    {System.out.println("");}
    ;

Protocoltest.java

package com.javadude.antlr3.x.tutorial;  
import org.antlr.runtime.*;  
import org.antlr.runtime.tree.*;  
import org.antlr.runtime.tree.CommonTree;  
import org.antlr.runtime.tree.CommonTreeNodeStream;  
public class Protocoltest { 



/**
 * @param args
 */
public static void main(String[] args) throws Exception {
    //create input stream from standard input
    ANTLRInputStream input = new ANTLRInputStream(System.in);
    //create a lexer attached to that input stream
    protocolLexer lexer = new protocolLexer(input);
    //create a stream of tokens pulled from the lexer
    CommonTokenStream tokens = new CommonTokenStream(lexer);

    //create a pareser attached to teh token stream
    protocolParser parser = new protocolParser(tokens);
    //invoke the program rule in get return value
    protocolParser.program_return r =parser.program();
    CommonTree t = (CommonTree)r.getTree();
    //output the extracted tree to the console
    System.out.println(t.toStringTree());

    //walk resulting tree; create treenode stream first
    CommonTreeNodeStream nodes = new CommonTreeNodeStream(t);
    //AST nodes have payloads that point into token stream
    nodes.setTokenStream(tokens);


    //create a tree walker attached to the nodes stream  
            //Error, can't create TreeGrammar object called walker
    protocolWalker walker = new protocolWalker(nodes);

    //invoke the start symbol, rule program
    walker.program();
    }
}

Problems:

1. In protocolWalker, I can't access the tokens (protocol.tokens)

   //Error, eclipse can't access tokenVocab named protocol  
       tokenVocab = protocol; //import tokens from protocol.g i.e, from protocol.tokens file
   

2. In In protocolWalker, can I create the object of java class, called Configuration, in the action list?

   protocol
       :^(PROT ID)
          {System.out.println("create protocol " +$ID.text);
           Configuration conf = new Configuration();
          }
       ;
   

3. In Protocoltest.java

   //create a tree walker attached to the nodes stream    
   //Error, can't create TreeGrammar object called walker  
       protocolWalker walker = new protocolWalker(nodes);  
   

   Object of protocolWalker can't be created. I have seen in the examples and the tutorials that such object is created.

Answer 1

In protocolWalker, I can't access the tokens (protocol.tokens)...

It seems to be accessing protocol.tokens fine: changing tokenVocab to something else produces an error that it doesn't produce now. The problem with protocolWalker.g is that it's defined as a token parser (grammar protocolWalker) but it's being used like a tree parser. Defining the grammar as tree grammar protocolWalker took away the errors that I was seeing about the undefined tokens.

In protocolWalker, can I create the object of java class, called Configuration, in the action list?

Yes, you can. The normal Java programming caveats apply about importing the class and so on, but it's as available to you as code like System.out.println.

In Protocoltest.java ... Object of protocolWalker can't be created.

protocolWalker.g (as it is now) produces a token parser named protocolWalkerParser. When you change it to a tree grammar, it'll produce a tree parser named protocolWalker instead.

Thanks a lot for posting the whole grammars. That made answering the question much easier.

Answer 2

Thank you for your reply, that was a silly mistake. Tokens problem and creating object of protocolWalker is resolved now but whenever, I change the grammar whether, protocol.g or protocolWalker.g, I had to write package name again(every time) in protocolParser.java and protocolWalker.java. I had the same problem with lexer file before but that was overcomed by the following declaration.

@header {
package com.javadude.antlr3.x.tutorial;
}

but I don't know how to overcome this problem?

Also, I have developed a GUI in Java using SWING where I have a textarea. In that text area, user will write the input, like for my grammar user will write,

protocol test;
pin = 5;
p = 3;
transitions = 2(5,0) (5,1);

How can I process this input in Java Swing GUI and produce output there?

Moreover, if I give the following section of protocolWalker.g to

protocol
    :^(PROT ID)
    {
     System.out.println("create protocol " +$ID.text);
     Configuration conf = new Configuration();
     conf.showConfiguration();
    }
    ;

initialprocess
    :^(INITIALP INT)
    {System.out.println("create initial process (with state) ");}
    ;

process
    :^(PROC INT)
    {System.out.println("create processes ");}
    ;

and run the test file with the following input,

protocol test;
pin = 5;
p = 3;
transitions = 2(5,0) (5,1);

I get the following output

(PROT test) (INITIALP 5) (PROC 3) (TRANSITIONS 2 5 0 5 1)
create protocol test

why the second and the third println in the protocolWalker.g are not shown in the output?

Any thoughts/help?

Thank you once again.