Antlr v3 error with parser/lexer rules

Question

I am having problems with my Antlr grammar. I'm trying to write a parser rule for 'typedident' which can accept the following inputs:

'int a' or 'char a'

The variable name 'a' is from my lexer rule 'IDENT' which is defined as follows:

IDENT : (('a'..'z'|'A'..'Z') | '_') (('a'..'z'|'A'..'Z')|('0'..'9')| '_')*;

My 'typedident' parser rule is as follows:

typedident : (INT|CHAR) IDENT;

INT and CHAR having been defined as tokens.

The problem I'm having is that when I test 'typedident' the variable name has to be more than one character. For example:

'int a' isn't accepted while 'int ab' is accepted.

The outputed error I get is:

"MismatchedTokenException: mismatched input 'a' expecting '$'"

Any idea why I'm getting this error? I'm fairly new to Antlr so apologies if the error is trivial.

EDIT

I literally just got it working, and I don't know why. I also had two other lexer rules defined as follows:

ALPH : ('a'..'z'|'A'..'Z'); 
DIGIT : ('0'..'9');

I realised these weren't being used at all so I deleted them and everything now works fine! My guess why this works is because ALPH and DIGIT were overriding my other Lexer rules:

NUMBER : ('0'..'9')+; 
CHARACTER : '\'' (~('\n' | '\r' |'\'')) '\'';

Does anyone know if this is the case? I'm curious as to why this problem has now been solved.

I literally just got it working, and I don't know why. I also had two other lexer rules defined as follows: ALPH : ('a'..'z'|'A'..'Z'); DIGIT : ('0'..'9'); I realised these weren't being used at all so I deleted them and everything now works fine! My guess why this works is because ALPH and DIGIT were overriding my other Lexer rules: NUMBER : ('0'..'9')+; CHARACTER : '\'' (~('\n' | '\r' |'\'')) '\''; Does anyone know if this is the case? I'm curious as to why this problem has now been solved. — Dan_Dan_Man

Bart Kiers Bart Kiers · Accepted Answer · 2012-02-12T18:36:28

'int a' isn't accepted while 'int ab' is accepted. ... My guess why this works is because ALPH and DIGIT were overriding ...

Yes, it appears ALPH was defined before the IDENT rule, in which case single letters were tokenized as ALPH tokens. If IDENT was defined before ALPH, it should all go okay (in your case).

To summarize how ANTLR's lexer rules work:

lexer rules match as much characters as possible (greedy);
if 2 (or more) lexer rules match the same input, the rule defined first will "win".

You must realize that the lexer does not produce tokens based on what the parser (at that time) needs. The lexer operates independently from the parser.

Antlr v3 error with parser/lexer rules

1 Answers