I have starting learning to write a lexer in ANTLR 4.5. From this page, which serves as documentation, I see that the following Lexer commands exist : more, pushMode(x), popMode, type(x), channel(x), mode(x), skip.
I have not been able to clearly understand their function. The following is my understanding of what each of them does:
skip
This skips all the characters that have been read in the current token. The past tokens are left untouched.
So, if the lexer has read some character a, and it next reads b, corresponding to
SOME_RULE : 'b' -> skip;
then it will throw away both a and b and go to the next token.
more
I am not sure what this does. The documentation says that the text that has been read will not be thrown away, but nothing about what tokens will finally be there. Suppose I have
RULE_1 : 'a' -> more;
RULE_2 : 'b';
If an a is read, and then a b, will the resulting token correspond to RULE_2 with a semantic value ab, or RULE_1 RULE_2, or something else?
type(x)
If I have
RULE_1 : 'a' -> type(TOKEN_1);
RULE_2 : 'b';
will only a be taken as the semantic value of TOKEN_1, or will all the characters corresponding to rules that were not tokens, right from the last token, be taken as the semantic value? If a b and an a arrive, will TOKEN_1s value be a or ba?
mode(x)
This switches the mode to a new mode. But here, are the characters read till the point of switch kept or discarded? What about tokens? Does each mode have a separate stack?
pushMode(x)
How is this different from mode? Is it the case that the already read characters are pushed to the mode it is going to, which doesn't happen in mode?
popMode
What is popped out? If I have
RULE_1 : 'a' -> popMode;
RULE_2 : 'b' ;
If I get a b and an a, will popMode cause only a to be returned, or ba, or the tokens, if any? Why do I occasionally get an error due to an empty stack?
channel(x)
I could not find an explanation for this. What is a channel?
Could anyone please clarify on the function of each of these commands, if possible with examples?
Please let me know if this is too broad.
Thank you.
