Antlr Using Parser Inside A Lexer

Question

I am new to Antlr and I have defined a basic grammar using Antlr 3.

The grammar can be seen below:

grammar data;

@header {
package com.data.language;
}


null        : 'null';
data        : null | STRING | INTEGER;
STRING  : ('a'..'z'|'A'..'Z')+;
INTEGER : '0'..'9'+;

This works perfectly and ANTLRWorks generates the code. However, I would like to use the data keyword within the lexer. For example:

I : data '*' INTEGER;

The problem is that once this is done, I am getting the following Exception:

Error 106: reference to undefined value: data

and I cannot see the data within the graph of ANTLRWorks. I can see only * INTEGER of the I lexer.

Is what I am trying to do, is it possible? How is it possible?

Sergey Kalinichenko Sergey Kalinichenko · Accepted Answer · 2012-11-21T15:28:10

Using parser productions in the lexer is not possible: lexer produces tokens ("words") that are later consumed by the parser, while parsers recognize sequences of words ("phrases"). The lexer layer is below the parser layer; there is no "feedback channel" that would let the lexer "learn" of what the parser is doing.

A parser rule like this

data_star_int : data '*' INTEGER;

is, obviously, fine, because parser rules can refer to both lexer tokens and other parser rules. You can use the data_star_int to match multi-token sequences.

I would replace the inlined * with a named token, but that is a matter of preference.

Antlr Using Parser Inside A Lexer

1 Answers