Handle hidden channel in antlr 3

Question

I am writing an ANTRL grammar for translating one language to another but the documentation on using the HIDDEN channel is very scarce. I cannot find an example anywhere. The only thing I have found is the FAQ on www.antlr.org which tells you how to access the hidden channel but not how best to use this functionality. The target language is Java.

In my grammar file, I pass whitespace and comments through like so:

// Send runs of space and tab characters to the hidden channel.        
WHITESPACE 
    :   (SPACE | TAB)+ { $channel = HIDDEN; }
    ;

// Single-line comments begin with --
SINGLE_COMMENT 
    :    ('--' COMMENT_CHARS NEWLINE) {
            $channel=HIDDEN;
        }
    ;

fragment COMMENT_CHARS 
    :   ~('\r' | '\n')*
    ;

// Treat runs of newline characters as a single NEWLINE token.
NEWLINE 
    :   ('\r'? '\n')+ { $channel = HIDDEN; }
    ;

In my members section I have defined a method for writing hidden channel tokens to my output StringStream...

@members {

private int savedIndex = 0;

void ProcessHiddenChannel(TokenStream input) {      
    List<Token> tokens = ((CommonTokenStream)input).getTokens(savedIndex, input.index());
    for(Token token: tokens) {
        if(token.getChannel() == token.HIDDEN_CHANNEL) {
            output.append(token.getText());

        }
    }
    savedIndex = input.index();
}
}

Now to use this, I have to call the method after every single token in my grammar.

myParserRule
        :       MYTOKEN1 { ProcessHiddenChannel(input); }
                MYTOKEN2 { ProcessHiddenChannel(input); }
        ;

Surely there must be a better way?

EDIT: This is an example of the input language:

-- -----------------------------------------------------------------
--
--
--  Name                Description
--  ==================================
--  IFM1/183         Freq Spectrum Inversion
--                     
-- -----------------------------------------------------------------

PROCEDURE IFM1/183

TITLE "Freq Spectrum Inversion";

HELP
      Freq Spectrum Inversion

ENDHELP;

PRIVILEGE CTRL;

WINDOW MANDATORY;

INPUT

   $Input : @NO_YES
   DEFAULT select %YES when /IFMS1/183.VALUE = %NO;
                  %NO otherwise
           endselect
   PROMPT "Spec Inv";

   $Forced_Cmd : BOOLEAN
   Default FALSE
   Prompt  "Forced Commanding";

DEFINE
   &RetCode   : @PSTATUS := %OK;
   &msg       : STRING;
   &Input     : BOOLEAN;

REQUIRE AVAILABLE(/IFMS1)
        MSG "IFMS1 not available";

REQUIRE /IFMS1/001.VALUE = %MON_AND_CTRL
        MSG "IFMS1 not in control mode";

BEGIN  -- Procedure Body --

    &msg := "IFMS1/183 -> " + toString($Input) + " : "; 

-- pre-check

   IF /IFMS1/183.VALUE = $Input
      AND $Forced_Cmd = FALSE THEN
      EXIT (%OK, MSG &msg + "already set");
   ENDIF;

-- command

   IF $Input =  %YES THEN &Input:= TRUE;
    ELSE  &Input:= FALSE;
   ENDIF;

   SET &RetCode := SEND IFMS1.FREQPLAN
        ( $FreqSpecInv := &Input);
   IF &RetCode <> %OK THEN
      EXIT (&RetCode, MSG &msg + "command failed");
   ENDIF;

-- verify

   SET &RetCode := VERIFY /IFMS1/183.VALUE = $Input TIMEOUT '10';
   IF &RetCode <> %OK THEN
      EXIT (&RetCode, MSG &msg + "verification failed");
   ELSE
      EXIT (&RetCode, MSG &msg + "verified");
   ENDIF;

END

Could you post an example of the language you're translating? (and also the final result of this translation) — Bart Kiers
The ANTLR Reference book shows examples with "rewrite mode" -- options {rewrite=true;} -- and a subclass of CommonTokenStream, TokenRewriteStream. I haven't used these, but they might be of interest. — Andy Thomas

Glenn Glenn · Accepted Answer · 2010-11-20T19:17:51

Look into inheriting CommonTokenStream and feeding an instance of your subclass into ANTLR. From the code example that you give, I suspect that you might be interested in taking a look at the filter and the rewrite options available in version 3.

Also, take a look at this other related stack overflow question.

Handle hidden channel in antlr 3

2 Answers