lexing multiline define statements with antlr4

Question

I am trying to write a lexer to do preprocessing, which can handle multi-line #define statements. For example, the following input, where the multi-line definition is broken by an subsequent empty line (may contain white spaces though):

aa(bb);
#define XX pqr
#define YY pqr \
    +abc

class p(XX,YY,zz);
endclass

The first step is to tokenize the input stream, where for any definition the value will be obtained as one string token. eg, for YY, I am trying to get "pqr+abc" as its string value. I have written the following lexer for tokenizing:

DEF: '#define' -> pushMode(def_mode);
ID: Letter (Letter | DecDigit)* ;
COMMENT : '/*' .*? '*/' -> channel(HIDDEN) ;
LINE_COMMENT : '//' ~('\n'|'\r')* NL  -> channel(HIDDEN);
WS: ( ' ' |'\t' | NL )+     -> channel(HIDDEN) ;
SEMICOLN: ';' ;
COMMA: ',' ;
OB: '(' ;
CB: ')' ;
PLUS: '+' ;
fragment NL : '\r'? '\n' ;
fragment DecDigit: '0'..'9' ;
fragment Letter: 'A'..'Z' | 'a'..'z' | '_' ;
mode def_mode;
    STR2: '\r'? '\n' -> popMode;
    STR1: ~('\n'|'\r')* '\r'? '\n' ;

The above lexer gives the following tokens for the #define lines:

[@10,26:32='#define',<2>,6:0]
[@11,33:42=' YY pqr \\n',<13>,6:7]
[@12,43:51='    +abc\n',<13>,7:0]
[@13,52:52='\n',<12>,8:0]
[@14,53:57='class',<3>,9:0]

The above tokens are obtained only if there is an "empty" line after the #define lines. If there are some whitespace in that line, ie it is not really empty, then the mode is not exited. Here are the tokens when the line has whitespace:

[@10,26:32='#define',<2>,6:0]
[@11,33:42=' YY pqr \\n',<13>,6:7]
[@12,43:51='    +abc\n',<13>,7:0]
[@13,52:55='   \n',<13>,8:0]
[@14,56:74='class p(XX,YY,zz);\n',<13>,9:0]
[@15,75:83='endclass\n',<13>,10:0]
[@16,84:84='\n',<12>,11:0]
[@17,85:84='<EOF>',<-1>,12:0]

Also, the lexer is not joining the two lines. How to fix these errors?

Is it possible to add a code to manually scan the input till an empty/blank line is seen, and then exit the mode? — R71

BernardK BernardK · Accepted Answer · 2017-02-06T13:54:05

I am not an expert (one grammar per year) and not a fan of mode, and don't like to make too much processing in the lexer. The parser has much more capabilities. Look at this :

grammar Question;

/* Parsing preprocessor #define */

program
    :   statement+
    ;

statement
    :   aClass
    |   function
    |   preprocessor
    ;

aClass
    :   'class'
        classDef // classBody
        'endclass'
    ;

classDef
    :   ID '(' list ')' ';'
    ;

function
    :   ID '(' list ')' ';'
    ;

preprocessor
    :   DEFINE ID replacement
        {System.out.println($DEFINE.text + " value=" + $ID.text + " -> replaced by " + $replacement.text);}
    ;

replacement
    :   expr+
    ;

expr
    :   ID
    |   ID '+' ID
    ;

list
    :   ID ( ',' ID )*
    ;

ID  :   LETTER ALPHAMERIC* ;
DEFINE
    :   '#' 'define' ;
COMMENT
    :   '/*' .*? '*/' -> channel(HIDDEN) ;
LINE_COMMENT
    :   '//' ~('\r' | '\n')* -> channel(HIDDEN) ;
WS  :   [ \t\r\n]+ -> channel(HIDDEN) ; // keep spaces in $<rule>.text
//WS  :     [ \t\r\n]+ -> skip ;
CONTINUATION
// if you want to keep the exact value including NL :
//  :   '\\' '\r'? '\n' -> channel(HIDDEN) ;
// to discard the continuation character :
    :   '\\' '\r'? '\n' -> skip ; // ignored as in Ruby, concatenates two lines

fragment LETTER     : [a-zA-Z_] ;
fragment DIGIT      : [0-9] ;
fragment ALPHAMERIC : LETTER | DIGIT ;

With the input data.txt :

/* function
        call */
aa(bb);
#define XX pqr
#define YY long replacement value
// multiline :
#define ZZ stu \
    +abc

class p(XX,YY,zz);
endclass
#define WW vwx \
    +def

// preceding line contains 10 spaces

$ hexdump -C data.txt 
...
000000b0  2b 64 65 66 0a 20 20 20  20 20 20 20 20 20 20 0a  |+def.          .|
000000c0  2f 2f 20 70 72 65 63 65  64 69 6e 67 20 6c 69 6e  |// preceding lin|
000000d0  65 20 63 6f 6e 74 61 69  6e 73 20 31 30 20 73 70  |e contains 10 sp|
000000e0  61 63 65 73                                       |aces|
000000e4

the output is :

$ grun Question program -tokens data.txt 
[@0,0:26='/* function\n        call */',<COMMENT>,channel=1,1:0]
[@1,27:27='\n',<WS>,channel=1,2:15]
[@2,28:29='aa',<ID>,3:0]
...
[@8,36:42='#define',<DEFINE>,4:0]
[@9,43:43=' ',<WS>,channel=1,4:7]
[@10,44:45='XX',<ID>,4:8]
[@11,46:46=' ',<WS>,channel=1,4:10]
[@12,47:49='pqr',<ID>,4:11]
[@13,50:50='\n',<WS>,channel=1,4:14]
...
#define value=XX -> replaced by pqr
#define value=YY -> replaced by long replacement value
#define value=ZZ -> replaced by stu     +abc
#define value=WW -> replaced by vwx     +def

If you use the skip version of WS, you'll have pqr+abc but also longreplacementvalue. I leave this point to your astuteness.

lexing multiline define statements with antlr4

1 Answers