Antlr4 override text of Lexer

Question

I know I can override the text of (parser) rules in many ways (listener or visitor).

However I want to manipulate the text given for some particular Lexical matches (Lexer rules).

Let's say that we work with java's grammar. And we have a complete list of Keywords:

ABSTRACT      : 'abstract';
ASSERT        : 'assert';
BOOLEAN       : 'boolean';
BREAK         : 'break';
// 50 more or so...

This is a sampling input:

public \t\t\t\t class Yolo{}

It's more convenient to skip the whitespace & comments instead of having them between parser rules, for sure. However when I create my translator I want have after each keyword a single whitespace:

public class Yolo{}

My huge problem is that it's really cumbersome to add in the Listener or Visitor the white space, so I was thinking if I could override a generic method that prints the text of the Lexer. Like:

@lexer::members {

    //the list of Keyword that I want them to be followed by a space
    ArrayList<Int> keyword = .... 

    @Override
    public String getText() {
        String text = super.getText();
        if( keywords.contains( getToken().getType()) ){
            text = text + " ";
        }
        return text;
    }
}

Update

The following should have worked because the source code handles the text override. I think that _text gets updated or reseted (with Lexer's reset() ) so my custom text never reaches the output.

@Override
public void emit(Token t) {
    super.emit(t);
    if(t.getType()==PACKAGE){
        setText(getText()+" ");
    }
}

Source:

/** Return the text matched so far for the current token or any
 *  text override.
 */
public String getText() {
    if ( _text !=null ) {
        return _text;
    }
    return getInterpreter().getText(_input);
}

I would not modify the lexer but rather change the way you get the text. In C# you could use extension methods to get the text with an extra space. In other languages it's at least possible to create a function that accepts a token an returns the modified text. — Onur

Diolor Diolor · Accepted Answer · 2014-09-07T18:19:25

Not sure if this is the best way but it works:

@Override
public Token emit() {
    if(getType()==PACKAGE){
        setText(getText()+" ");
    }
    return super.emit();
}

I overrided public Token emit() instead of public void emit(Token token).

Antlr4 override text of Lexer

1 Answers