ANTLR mismatched input ?wrong lexer order?

Question

I have started working with Antlr to generate a parser for a self defined DSL. The language takes a file and filters it.

My grammar looks like this (shortened):

grammar ParserGrammar;
filter              :   FILTER' '(TYPE|INSTANCE) (((' 'SPECIFIER)+)|((' 'SPECIFIER)*'         'EXCLUDE(' 'SPECIFIER)+))' '(WHERE condition)?;
condition           :   (attributecondition|propertycondition|quantitycondition)+ ('&&' condition)?;
attributecondition  :   ATTRIBUTE SPECIFIER comparisonoperation SPECIFIER;
propertycondition   :   PROPERTY SPECIFIER SPECIFIER comparisonoperation SPECIFIER;
quantitycondition   :   QUANTITY SPECIFIER comparisonoperation SPECIFIER;
comparisonoperation :   (EQUALS|GREATER|LOWER|GREATEREQ|LOWEREQ);

check               :   CHECK (entitycheck|relationcheck) (checkoperation COUNT)?;
entitycheck         :   ENTITY SPECIFIER* (HAVING (attributecheck|quantitycheck|propertycheck))?;
attributecheck      :   ATTRIBUTE SPECIFIER;
quantitycheck       :   QUANTITY SPECIFIER;
propertycheck       :   PROPERTYSET SPECIFIER (HAVING PROPERTY SPECIFIER)?;
relationcheck       :   RELATION SPECIFIER+ WITH SPECIFIER+;
checkoperation      :   '-'(EQUALS|GREATER|LOWER|GREATEREQ|LOWEREQ);

FILTER      :   F I L T E R;
CHECK       :   C H E C K;
TYPE        :   '-'T Y P E;
INSTANCE    :   '-'I N S T A N C E;
EXCLUDE     :   '-'E X C L U D E;
WHERE       :   '-'W H E R E;
ATTRIBUTE   :   A T T R I B U T E;
PROPERTY    :   P R O P E R T Y;
QUANTITY    :   Q U A N T I T Y;
EQUALS      :   E Q U A L S;
GREATER     :   G R E A T E R;
LOWER       :   L O W E R;
GREATEREQ   :   GREATER E Q;
LOWEREQ     :   LOWER E Q;
ENTITY      :   E N T I T Y;
RELATION    :   R E L A T I O N;
HAVING      :   '-'H A V I N G;
PROPERTYSET :   PROPERTY S E T;
COUNT       :   Num+('.'Num+)?;
WITH        :   '-'W I T H;
SPECIFIER   :   ('"'(~'"')+'"')|(~'"')+;


fragment Num :  ('0'..'9');
fragment A:('a'|'A');
//skipped here
fragment Z:('z'|'Z');

So lets say I want to parse this input: filter -type abc

Then I get:

mismatched input 'filter -type abc ' expecting FILTER

The thing is, if I chance the lexer rule SPECIFIER to

SPECIFIER : 'abc'

and I type: filter -type abc it works just fine.

I believe it is a problem with the lexer ordering in connection with the greedy matching? Even if I don't understand why, because I thought Antlr takes the first matching lexer rule. And in my case, the rules FILTER and TYPE are in front of SPECIFIER.

Thanks in advance,

Stephan

Lucas Trzesniewski Lucas Trzesniewski · Accepted Answer · 2014-09-08T19:54:55

I thought Antlr takes the first matching lexer rule. And in my case, the rules FILTER and TYPE are in front of SPECIFIER.

Well... this is not entirely true. ANTLR takes the longest matching rule first, and only if several rules match the same token length will it choose the one that appears first.

Your SPECIFIER rule is too broad and matches too much text, try to narrow it. In general, it's better to use inclusive rules than exclusive ones IMHO, exactly because of this matching strategy.

ANTLR mismatched input ?wrong lexer order?

1 Answers