Island Solution with ANTLR4

Question

I'd like to share with you a island solution I had to implement in ANTLR4.

1. Structure of the language. The language I had to write the grammar for is derived from PL/SQL with some additional constructs. I won't go into more details here as this is off topic.

2. The language defines a special command PUT with the following structure:

   PUT [<SPECIALISED LANGUAGE>].

My solution was:

1. Override Lexer's nextToken method:

   public Token nextToken() {
       if (f_current_idx != -1) {
           _input.seek(f_current_idx); f_current_idx = -1;
       }
       Token l_token = super.nextToken(); return l_token;
   }
   

2. Add some code in the Lexer:

   PUT :
   'PUT'
   {
       f_current_idx = _input.index(); ((ANTLRStringStream) _input).rewind();
   
       SRC_PUTLexer l_put_lexer = new SRC_PUTLexer(_input);
       UnbufferedTokenStream<Token> l_tokenStream = new UnbufferedTokenStream<Token>(l_put_lexer);
   
       if (l_tokenStream.LA(2) == SRC_PUTLexer.LBRACK) { 
           new SRC_PUTParser(l_tokenStream).start_rule(); f_current_idx = _input.index();
       } 
   };
   

Furthermore the class ANTLRStringStream which has disappeared in ANTLR 4 had to be defined:

public class ANTLRStringStream extends ANTLRInputStream {

    protected int markDepth = 0;
    protected int lastMarker;

    protected ArrayList<Integer> markers;

    public ANTLRStringStream() {
        super();
    }

    public ANTLRStringStream(String input) {
        super(input);
    }

    public int mark() {
        if ( markers==null ) { markers = new ArrayList<Integer>(); }
        markers.add(markDepth, index()); markDepth++; lastMarker = markDepth;
        return markDepth;
    }

    public void rewind(int m) {
        int state = (int) markers.get(m); seek(state); release(m);
    }

    public void rewind() { rewind(lastMarker); }

    public void release(int marker) {
        markDepth = marker; markDepth--;
    }
}

Any feedback would be very welcome! Kind regards, Wolfgang Hämmer

Answer 1

This should really be a community wiki.

My first major comment is you need to get rid of the ANTLRStringStream class. The ANTLRInputStream class provided by ANTLR 4 provides the functionality of ANTLRStringStream in ANTLR 3. The IntStream and CharStream interfaces were revised and extensively documented in ANTLR 4 to get rid of the problematic rewind methods and other undefined behavior. You should not reintroduce them.