Migration tool for ANTLR grammar

Question

Suppose I have a following simple grammar (query DSL):

grammar TestGrammar;

term : textTerm ;

textTerm : 'Text' '(' T_VALUE '=' STRING+ ')' ;

T_VALUE : 'value' ;
STRING : '"' .+? '"' ;

WS : [ \t\r\n]+ -> skip ;

Then at some point I decide that text term format needs to be changed, for example:

Text(value = "123") -> MyText(val = "123")

How should I approach migrating existing data that users have generated with previous version of grammar?

For simple changes like these, TokenStreamRewriter is exactly what you need. — Lucas Trzesniewski
Thanks, @LucasTrzesniewski. I finally managed to try this and it seems to do the trick. — Andrew Logvinov

quepas quepas · Accepted Answer · 2016-11-13T19:06:42

Assumption

Let's make one simplification of your grammar by introducing token TEXT for 'Text' string.

grammar TestGrammar;

WS : [ \t\r\n]+ -> channel(HIDDEN);  // preserve the whitespaces characters!
T_VALUE : 'value';
STRING : '"' .+? '"';
TEXT : 'Text';

term
    : textTerm;
textTerm
    : TEXT '(' T_VALUE '=' STRING+ ')';

Solution

Now we will utilize the AST listener built by ANTLRv4 tool. This allows us to traverse AST and perform token replacement with TokenStreamRewriter class already mentioned by Lucas Trzesniewski in comments.

import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.TokenStreamRewriter;

public class MigrationTask extends TestGrammarBaseListener {
    private TokenStreamRewriter rewriter;

    public MigrationTask(CommonTokenStream stream) {
        this.rewriter = new TokenStreamRewriter(stream);
    }

    @Override
    public void enterTextTerm(TestGrammarParser.TextTermContext ctx) {
        rewriter.replace(ctx.TEXT().getSymbol(), "MyText");
        rewriter.replace(ctx.T_VALUE().getSymbol(), "val");
    }

    public String getMigrationResult() {
        return rewriter.getText();
    }
}

So, we substitute given token with its replacement whenever we encounter the token during the traversal of AST.

Usage

Now we can execute MigrationTask on given ParseTree and retrive the migration result:

(...)
CommonTokenStream tokens = new CommonTokenStream(lexer);
TestGrammarParser parser = new TestGrammarParser(tokens);
ParseTree tree = parser.term();
ParseTreeWalker walker = new ParseTreeWalker();
MigrationTask migrationTask = new MigrationTask(tokens);
walker.walk(migrationTask, tree);
String result = migrationTask.getMigrationResult(); // Here we retrive migration result !
(...)

Migration tool for ANTLR grammar

1 Answers

Assumption

Solution

Usage