Antlr4 strange parsing behavior

Question

grammar Simpletest;
prog: test_statement* EOF ;

test_statement
  :COPY_TABLE ID INTO_WORD ID    # copytable;


ID
    : (SIMPLE_LETTER) (SIMPLE_LETTER | '$' | '_' | '#' | ('0'..'9'))*;

COPY_TABLE: 'COPY TABLE';

SIMPLE_LETTER
    : 'a'..'z'
    | 'A'..'Z'
    ;

INTO_WORD: 'INTO';

SPACES
    : [ \t\n\r]+ -> skip
    ;
NEWLINE : [\r\n]+ ;

As you can see in the above test grammar, I would expect it to parse something like "COPY TABLE blah INTO blah2" rather easily. However, the result is surprising.

$ grun Simpletest prog -tokens -trace -diagnostics -tree
COPY TABLE blah INTO blah2
[@0,0:9='COPY TABLE',<2>,1:0]
[@1,11:14='blah',<1>,1:11]
[@2,16:19='INTO',<1>,1:16]
[@3,21:25='blah2',<1>,1:21]
[@4,27:26='<EOF>',<-1>,2:0]
enter   prog, LT(1)=COPY TABLE
enter   test_statement, LT(1)=COPY TABLE
consume [@0,0:9='COPY TABLE',<2>,1:0] rule test_statement
consume [@1,11:14='blah',<1>,1:11] rule test_statement
line 1:16 missing 'INTO' at 'INTO'
consume [@2,16:19='INTO',<1>,1:16] rule test_statement
exit    test_statement, LT(1)=blah2
line 1:21 extraneous input 'blah2' expecting {<EOF>, 'COPY TABLE'}
consume [@4,27:26='<EOF>',<-1>,2:0] rule prog
exit    prog, LT(1)=<EOF>

It matched COPYTABLE and blah successfully, before failing to match INTO with INTO!!

Could someone explain why this might be happening?

royalflush royalflush · Accepted Answer · 2016-09-07T06:37:32

Well, after staring at my own question for one minute, I figured it out. It has to do with ambiguous lex rules. 'INTO' could be matched with both INTO_WORD and ID. Since ID is at the top of the lexer rules, it matches first.

Antlr4 strange parsing behavior

1 Answers