antlr how can I skip a token until I need it

Question

Is there a way to skip tokens until I need them? In order to be more clear, here is a grammar that is as close as I could get to what I want:

grammar example;

file : statement* EOF ;
statement : ID EOL
          | '{' (EOL statement*)? '}' EOL
          ;

EOL : ('\r'? '\n' | '\r') -> skip ;
WHITESPACE : [ \t]+ -> skip ;

Hopefully my intent is clear: all whitespace (including newlines) is skipped under normal circumstances, but I can demand the presence of a newline whenever I want, so

foo
{
  bar
}
baz

would fit the grammar, but not

foo {
  bar
} baz

or

foo bar
{
  baz
}

Is there a way to do this, or do I just have to put a lot of EOL*'s in my grammar?

What is the semantic purpose behind 1)"skipping a token" and 2) imposing whitespace requirements on the grammar? It might help if you could explain those requirements, because they are unique in my experience. — TomServo
@TomServo the point of skipping a token is so that it's not gathered into tokens (or at least, those tokens are discarded), and the point of imposing whitespace would be for something like Swift, where under normal circumstances all whitespace is ignored (so a = b * 2 is the same as a = b* 2) but you can say that the end of a statement can be either ; or a the end of the line, so you don't always have to write the ;. — Anonymous

Raven Raven · Accepted Answer · 2017-08-15T11:37:25

Not long ago I answered another question that needed the same kind of mechanism.
See here for further details.

Basically you achieve this by providing your own, custom TokenStream that implements a mechanism of either skipping whitespace or feeding it into the parser depending on it's setting.

antlr how can I skip a token until I need it

1 Answers