Why am I getting a NullPointerException when I import lexers in Antlr?

Question

I am using antlr 3 and Antlrworks. Here is my setup:

lexer Base //contains basic tokens - like WS, number etc.

lexer Specific //contains my language specific tokens - AND derives from Base lexer

parser specific //parser for my language

combined grammer -> imports specific lexer and specific parser

When I generate, I always get a NPE (in Java). The reason is that the reference to the Base lexer in the generated specific lexer is not initialized.

Am I missing something?

Bart Kiers Bart Kiers · Accepted Answer · 2011-11-17T16:35:18

Impossible to tell without seeing how you're importing the grammars.

Bear in mind that:

lexer grammars can only import other lexer grammars;
parser grammars can only import other parser grammars;
tree grammars can only import other tree grammars;
combined grammars can import lexer- and parser grammars (but not other combined grammars!).

In your case, that would look like:

BaseLexer.g

lexer grammar BaseLexer;

Num   : '0'..'9'+;
Space : ' ' | '\t';

SpecificLexer.g

lexer grammar SpecificLexer;

import BaseLexer;

SpecificTokenA : 'specificA';
SpecificTokenB : 'specificB';

SpecificParser.g

parser grammar SpecificParser;

specific :  SpecificTokenA |  SpecificTokenB;

Combined.g

grammar Combined;

import SpecificLexer, SpecificParser;

parse 
  :  Num Space specific EOF 
     {
       System.out.println("Parsed:\n  Num      = " + 
           $Num.text + "\n  specific = " + $specific.text);
     } 
  ;

and to test it all, use the class:

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    SpecificLexer lexer = new SpecificLexer(new ANTLRStringStream("42 specificB"));
    CombinedParser parser = new CombinedParser(new CommonTokenStream(lexer));
    parser.parse();
  }
}

Now generate the lexers and parsers and run the Main class:

java -cp antlr-3.3.jar org.antlr.Tool BaseLexer.g
java -cp antlr-3.3.jar org.antlr.Tool SpecificLexer.g
java -cp antlr-3.3.jar org.antlr.Tool SpecificParser.g
java -cp antlr-3.3.jar org.antlr.Tool Combined.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main

which will print the following to your console:

Parsed:
  Num      = 42
  specific = specificB

Also tested with ANTLRWorks 1.4.3.