Antlr lexer semantic predicate on an alternative

Question

Given the grammar:

grammar Test;
words: (WORD|SPACE|DOT)+;
WORD : (
       LD
       |DOT       {_input.LA(1)!='.'}?
       ) +        ;
DOT: '.';
SPACE: ' ';
fragment LD: ~[.\n\r ];

with Antlr4 generated Lexer, for an input:

test. test.test test..test

The token sequence is like:

[@0,0:4='test.',<1>,1:0]
[@1,5:5=' ',<3>,1:5]
[@2,6:14='test.test',<1>,1:6]
[@3,15:15=' ',<3>,1:15]
[@4,16:19='test',<1>,1:16]
[@5,20:20='.',<2>,1:20]
[@6,21:25='.test',<1>,1:21]
[@7,26:25='<EOF>',<-1>,1:26]

What puzzles why the last piece of text test..test is tokenized into test . and .test, while I was supposed to see test. .test

What puzzled me more is for input:

test..test test. test.test

the token sequence is:

[@0,0:3='test',<1>,1:0]
[@1,4:4='.',<2>,1:4]
[@2,5:9='.test',<1>,1:5]
[@3,10:10=' ',<3>,1:10]
[@4,11:14='test',<1>,1:11]
[@5,15:15='.',<1>,1:15]
[@6,16:16=' ',<3>,1:16]
[@7,17:20='test',<1>,1:17]
[@8,21:25='.test',<1>,1:21]
[@9,26:25='<EOF>',<-1>,1:26]

Here the test.test is separated into two tokens while in above it is one. Is the calling of _input.LA(1) has some side effect to cause this? Can some one explain?

I'm using Antlr4.

Am I right, that you want to ignore double DOT in your WORD lexer rule? — Cv4
yes. it is actually from my another question: :stackoverflow.com/questions/19224181/… I formalised the root problem to this question hoping to make it more clear. — Wudong

Cv4 Cv4 · Accepted Answer · 2013-10-10T09:29:19

Quick fix is to check the previous LA(-1) token if it is unequal . and add a leading optional DOT.

Resulting grammar is:

grammar Test;
words: (WORD|SPACE|DOT)+;
WORD : DOT? (
       LD
       |{_input.LA(-1)!='.'}? DOT       
       ) +        ;
DOT: '.';
SPACE: ' ';
fragment LD: ~[.\n\r ];

Have fun and enjoy ANTLR, it is a nice tool.

Antlr lexer semantic predicate on an alternative

1 Answers