How to rewrite token with text in tree grammar?

1
votes

This is my tree grammar:

grammar t;
options{
  output = AST;
}
type
  :
  'NVARCHAR' -> "VARCHAR"
  ;

ANTLR3 3.1.3 says:

syntax error: antlr: t.g:12:5: unexpected token: 'NVARCHAR'

What's wrong here? I took it from this article.

ps. I'm using this grammar later in order to get AST out of it. Once the AST is retrieved I'm walking through it and add every token's text to some string buffer. The idea of the rewriting above is to replace certain tokens. I'm doing language-to-language mapping (SQL to SQL dialect, to be more specific).

2
javaantlrantlr3

2 Answers

4
votes

Note the first sentence Terence starts with: "just had some cool ideas about a semantic rule specification language...". That's what the first example is: an idea. It's not valid syntax.

There are (at least) two options for you:

1. rewrite the text in the token immediately

grammar T;

options{
  output=AST;
}

@parser::members {
  public static void main(String[] args) throws Exception {
    TLexer lexer = new TLexer(new ANTLRStringStream("NVARCHAR"));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.type();
  }
}

type
  :  NVARCHAR {System.out.println("token=" + $NVARCHAR.text);}
  ;

NVARCHAR
  :  'NVARCHAR' {setText("VARCHAR");}
  ;

But this only adjusts the text, not the type of the token, which remains a NVARCHAR type.

2. use an imaginary token:

grammar T;

options{
  output=AST;
}

tokens {
  VARCHAR='VARCHAR';
}

@parser::members {
  public static void main(String[] args) throws Exception {
    TLexer lexer = new TLexer(new ANTLRStringStream("NVARCHAR"));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.type();
  }
}

type
  :  NVARCHAR -> VARCHAR
  ;

NVARCHAR
  :  'NVARCHAR'
  ;

which changes the text and type of the token.

As you can see, with both demos, token=VARCHAR is being printed to the console:

bart@hades:~/Programming/ANTLR/Demos/T$ java -cp antlr-3.3.jar org.antlr.Tool T.g
bart@hades:~/Programming/ANTLR/Demos/T$ javac -cp antlr-3.3.jar *.java
bart@hades:~/Programming/ANTLR/Demos/T$ java -cp .:antlr-3.3.jar TParser 
token=VARCHAR
1
votes

in antlr4 replacing text and type can be achieved with the type action: 

OldTokenType: 
    ('Token1' | 'Token2' | 'Token3' ) {setText("New Token");} 
    -> type(NewTokenType);