Collapsing whitespace in ANTLR4

1
votes

In my grammar, I have a whitespace token that is sent to the HIDDEN channel:

SP : [ \u00A0\u000B\t\r\n] -> channel(HIDDEN);

I know that I can get the text of a parsed rule, including hidden tokens, with TokenStream#getText(Context). I'd like to have all whitespace collapsed when I call that.

I also know there's a TokenStreamRewriter for rewriting specific tokens, but I don't see a way to rewrite all of a certain type.

Is there any way to collapse all SP tokens to output as a single space?

1
antlrtokenizeantlr4
At minimum, this is almost a duplicate of stackoverflow.com/q/25812615/138304. - Sam Harwell
Indeed. I asked this first, then realized there was a more general question hiding within; I felt it was different enough that it didn't feel appropriate to edit the entire question. - NickAldwin

1 Answers

1
votes

This can be done by changing the rule to match one or more whitespace characters (+) and specifying an action:

SP : [ \u00A0\u000B\t\r\n]+ { _text = " "; } -> channel(HIDDEN);