how to create token file from grammar file in antlr

0
votes

I am creating a DSL using antlr. Lexer and Parser are written in one grammar file(say layout.g). Tree grammar is written in another grammar file (say layoutTree.g). Now Tree parser is not properly parsing. I printed the AST output from parser, and its correct. I walked through the generated tree parser code, and found that token value declarations assign different values in tree parser and parser.Below is the sample output from parser and tree parser.

Parser output

public static final int ARRAY_MEMBER_TOKEN=4;
public static final int ARRAY_TOKEN=5;
public static final int DECLARATION_TOKEN=6;

Tree Parser Output

public static final int EOF=-1;
public static final int DECLARATION_TOKEN=4;
public static final int IDENTIFIER=5;

As you can see DECLARATION_TOKEN has different value in parser output and tree parser output. Because of this tree parser is not working as expected.How can I correct this problem?

Is it a problem with generated token file(say layout.token)? This file is empty in my project.How can I generate this file?

1
javaantlrantlr3antlrv3ide
A .tokens file is generated automatically while generating your lexer/parser(s). The fact that your .tokens file is empty, is not good. How are you generating your lexer/parser(s)? (command line, IDE, ANTLRWorks, other?) – Bart Kiers
I use 'antlrv3ide' from sourceforge.net. I have installed it from "antlrv3ide.sourceforge.net/updates". How can I generate it using command line? – hnm
On the command line, do: java -cp antlr-3.2.jar org.antlr.Tool layout.g and then java -cp antlr-3.2.jar org.antlr.Tool layoutTree.g – Bart Kiers

1 Answers

0
votes

Do you have something like this in your tree grammar? 

options
{
  tokenVocab=layout; //NOT layout.g or layout.tokens
  ASTLabelType=pANTLR3_BASE_TREE;
}