Grammar modularisation: A way to import lexer rules after defining others first?

Question

Say I have a simple grammar (lexer and parser) for parsing & evaluating simple mathematical expressions (as in a few antlr examples out there) that also allows simple variable definitions (i.e. assigning float values) and use of these variables. E.g. the following can be handled:

 r = 2.5;
 PI = 3.14;
 PI * r * r;

This is supposed to be used in a more complex grammar. In fact, several different ones. The problem is that the lexer for the above contains recognizes basically every string as token type ID, i.e. potential variable name, but the more complex grammars may contain other keywords.

If I do

lexer grammar ComplexLexer;
import SimpleMathExprLexer;
// ...
IF : 'if'|'IF';
THEN : 'then'|'THEN';
// ...

then it is not terribly helpful that ID already matches these keywords. Simply moving the import statement down below these rules does not work. Is there any way around this, or am I on the wrong path entirely when I look at composition?

This is a notoriously difficult problem in any setting (not just antlr --- just in grammars in general). One approach I'm aware of is described here: itu.dk/~brabrand/banana-algebra — Gian

Bart Kiers Bart Kiers · Accepted Answer · 2012-11-09T20:23:39

ANTLR will simply give precedence to the rule defined first. This means that if you have a lexer grammar G, that imports lexer grammars G3 and after that grammar G2:

lexer grammar G;

import G3, G2;

...

or similarly:

lexer grammar G;

import G3;

...

lexer grammar G3;

import G2;

...

the rules from G3 would get precedence over G2, but the rules from G would get precedence over both G3 and G2.

Just as with single lexer grammars, the rules like IDENTIFIER you'd define after rules that match keywords like "if", "then", ...

Grammar modularisation: A way to import lexer rules after defining others first?

1 Answers