ANTLR4: Unrecognized constant value in a lexer command

Question

I am learning how to use the "more" lexer command. I typed in the lexer grammar shown in the ANTLR book, page 281:

lexer grammar Lexer_To_Test_More_Command ;

LQUOTE : '"'        -> more, mode(STR) ;

WS   : [ \t\r\n]+   -> skip ; 

mode STR ;

STRING : '"'    -> mode(DEFAULT_MODE) ;

TEXT : .        -> more ;

Then I created this simple parser to use the lexer:

grammar Parser_To_Test_More_Command ;

import Lexer_To_Test_More_Command ;

test: STRING EOF ;

Then I opened a DOS window and entered this command:

antlr4 Parser_To_Test_More_Command.g4

That generated this warning message:

warning(155): Parser_To_Test_More_Command.g4:3:29: rule LQUOTE contains a lexer command with an unrecognized constant value; lexer interpreters may produce incorrect output

Am I doing something wrong in the lexer or parser?

Sam Harwell Sam Harwell · Accepted Answer · 2015-02-07T21:09:57

Combined grammars (which are grammars that start with just grammar, instead of parser grammar or lexer grammar) cannot use lexer modes. Instead of using the import feature¹, you should use the tokenVocab feature like this:

Lexer_To_Test_More_Command.g4:

lexer grammar Lexer_To_Test_More_Command;

// lexer rules and modes here

Parser_To_Test_More_Command.g4:

parser grammar Parser_To_Test_More_Command;

options {
  tokenVocab = Lexer_To_Test_More_Command;
}

// parser rules here

¹ I actually recommend avoiding the import statement altogether in ANTLR. The method I described above is almost always preferable.

ANTLR4: Unrecognized constant value in a lexer command

1 Answers