Writing parser rules sensitive to whitespace while skipping WS from the lexer

Question

I am having some troubles in handling whitespace. In the following excerpt of a grammar, I set up the lexer so that the parser skips whitespace:

ENTITY_VAR
    : 'user'
    | 'resource'
    ;

INT : DIGIT+ | '-' DIGIT+ ;
ID : LETTER (LETTER | DIGIT | SPECIAL)* ;
ENTITY_ID : '__' ENTITY_VAR ('_w_' ID)?;

NEWLINE : '\r'? '\n';

WS : [ \t\r\n]+ -> skip; // skip spaces, tabs, newlines

fragment LETTER : [a-zA-Z];
fragment DIGIT : [0-9];
fragment SPECIAL : ('_' | '#' );

The problem is, I would like to match against variables names of the form ENTITY_ID such that the matched string does not have any whitespace. It would be sufficient to write it as a lexer rule as I did here, but the thing is that I'd like to do it with a parser rule instead, because I want to have direct access to those two tokens ENTITY_VAR and ID individually from my code, and not squeeze them back together in a whole token ENTITY_ID.

Any ideas, please? Basically any solution which let me access directly ENTITY_VAR and ID would suit me, both by leaving ENTITY_ID as a lexer rule or moving it to the parser.

Perhaps lexical modes can help? Once you stumble upon '__', you switch modes where you don't skip spaces? — Bart Kiers
Thanks for your suggestion. So, if I understand correctly, I'd write a parser rule entityVar such that when matching against '__' it switches to a mode where the WS lexer rule is disabled? — Riccardo T.
Lexer modes are not dependent on parser rules. Whenever the lexer matches '__', it would switch modes, regardless if there is a parser rule that actually uses the '__' token. — Bart Kiers
Could you give some sample input code and what you'd like to have? — Onur

Onur Onur · Accepted Answer · 2014-07-31T09:45:25

There are several approaches I can think of (not in a special order):

Emit several tokens from the rule ENTITY_ID. See ANTLR4: How to inject tokens for an inspiration
Allow whitespace in the parser and check afterwards
Use the single token and split in code
Use the single token and modify the token stream before passing it to the parser. I.e. lex, modify the ENTITY_ID tokens and split them into several other tokens, then pass this stream to the parser
Don't skip whitespace and when dealing with these "extra tokens" check if they are within a ENTITY_ID part (=> is error) or not (=> ignore error).
Don't skip whitespace and add "WS*" everywhere in your grammar where whitespace is allowed (ok if the grammar is not too large).
Insert predicates in the parser rule that checks if there is whitespace between.

Create a "trap" rule like this:

INVALID_ENTITY_ID : '__' WS+ ENTITY_VAR WS? ('_w_' WS? ID)?
                  | '__' WS? ENTITY_VAR WS+ ('_w_' WS? ID)?
                  | '__' WS? ENTITY_VAR WS? ('_w_' WS+ ID)
                  ;

This will catch invalid ENTITY_IDs since it's longer than the parts that will then be also individual tokens.

I'd go with 2, if it doesn't alter the parse in the "non error" case, i.e. no code is interpreted differently by allowing whitespace.

Writing parser rules sensitive to whitespace while skipping WS from the lexer

3 Answers