ANTLR4 commonTokenStream.GetTokens() doesn't get all tokens

2
votes

I have an ANTLR4 grammar which is used in a method for syntax highlighting. When the user changes the text, the method is triggered and generates commonTokenStream. When I call GetTokens() I only get some of the expected tokens. When I call Consume(), I get some more but not all. How do I get all tokens in the text? 

AntlrInputStream stream = new AntlrInputStream(_input);
GrammarLexer lexer = new GrammarLexer(stream);
CommonTokenStream commonTokenStream = new CommonTokenStream(lexer);
List<IToken> list = commonTokenStream.GetTokens();
1
c#antlrantlr4

1 Answers

1
votes

Before commonTokenStream.GetTokens(), try a commonTokenStream.Fill() first:

CommonTokenStream commonTokenStream = new CommonTokenStream(lexer);
commonTokenStream.Fill();
List<IToken> list = commonTokenStream.GetTokens();