Can Antlr C parser recover from invalid tokens

Question

I developed an Antlr3.4, grammar which generates an AST for later parsing. The generated parser uses Antlr's C interface. When the parser encounters an unexpected token it adds

"Tree Error Node" to the AST token stream and continues on processing input. (Internally "Tree Error Node" represents ANTLR3_TOKEN_INVALID.)

When I pass the output of the parser to the AST parser, it halts upon the "Tree Error Node". Is there anyway to handle invalid tokens in an AST stream?

I'm using:

• libantlr3c-3.4
• antlr3.4

Answer 1

I turns out you can override the tree adaptor method "errorNode" to emit a user specified token. That token can then be handled in the AST parser.

Answer 2

You need to override Match() using the method described above and perform recover of the parser (this is c# pseudo code):

    public override object Match(IIntStream input, int ttype, BitSet follow)
    {
        if (needs recover)
        {
            ... Recover from mismatch, i.e. skip until next valid terminal.
        }

        return base.Match(input, ttype, follow);
    }

Also, you need to recover from mismatched token:

    protected override object RecoverFromMismatchedToken(IIntStream input, int ttype, BitSet follow)
    {
        if (needs recover)
        {
            if (unwanted token(input, ttype))
            {

                .. go to the next valid terminal
                .. consume as if ok
                .. return next valid token

            }

            if (missing token(input, follow))
            {
                .. go to the next valid terminal
                .. insert missing symbol and return
            }

            .. othwerwise throw
        }

        .. call base recovery(input, ttype, follow);
    }

Let me know if there are additional questions.