ANTLR parse multiple files to produce one AST

5
votes

How do i parse multiple source file and end up with just one AST to perform analysis and code generation from? Typically, I find example usage of ANTLR in the form of

public void process(String source) 
{
    ANTLRStringStream Input = new ANTLRStringStream(input);
    TLexer lex = new TLexer(Input);
    CommonTokenStream tokens = new CommonTokenStream(lex);
    TParser parser = new TParser(tokens);
    var tree = parser.parse().Tree; 
}

but neither the lexer nor the parser seems to be able to take additional files. Am I supposed to create a lexer and parser pr. inputfile and use tree.Add() to add trees from the other files to the tree of the first file?

2
antlr
What about reading all files into a string, and parsing that string? – Bart Kiers
Alas, then I have no way of returning sensible error messages with filename and linenumbers. – Carlo V. Dango

2 Answers

4
votes

Here are three ways you could do this:

  1. Use Bart's suggestion and combine the files into a single buffer. This would require adding a lexer rule that implements identical functionality to the C++ #line directive.

  2. Combine the trees returned by the parser rule.

  3. Use multiple input streams with a single lexer. This can be done by using code similar to that which handles include files by pushing all buffers onto the stack before lexing.

The second option would probably be the easiest. I don't use the Java target so I can't give code details, which is required for all of these solutions.

1
votes

I think this close to what you are after. I've hard-coded two files to process but you can process as many as needed by creating a loop. At the step // create new parent node and merge trees here into fulltree see Bart's answer on duplicating a tree. It has the steps to create a parent node and attach children to it (sorry but I've not done this and didn't have time to integrate his code and test). 

public class OneASTfromTwoFiles {
    public String source1 = "file1.txt";
    public String source2 = "file2.txt";

    public static void main(String[] args)
    CommonTree fulltree;
    {
        CommonTree nodes1 = process(source1);
        CommonTree nodes2 = process(source2);
        // create new parent node and merge trees here into fulltree
        CommonTreeNodeStream nodes = new CommonTreeNodeStream(fulltree); //create node stream
        treeEval walker = new treeEval(fulltree);
        walker.startRule(); //walk the combined tree
    }

    public CommonTree process(String source)
    {
        CharStream afs = null;
        // read file; exit if error
        try { 
            afs = new ANTLRFileStream(source);
        }
        catch (IOException e) {
            System.out.println("file not found");
            System.exit(1);
        }

        TLexer lex = new TLexer(afs);
        CommonTokenStream tokens = new CommonTokenStream(lex);
        TParser parser = new TParser(tokens);
        //note startRule is the name of the first rule in your parser grammar
        TParser.startRule_return r = parser.startRule(); //parse this file
        CommonTree ast = (CommonTree)r.getTree(); //create AST from parse of this file
        return ast; //and return it
    }
}