In ANTLR3, I need to discriminate between a 'comment' and a 'directive' (which looks like a comment)

1
votes

I'm fairly new to ANTLR, and I've run into a problem.

I have a grammar I'm trying to write for a language that includes single-line comments and language directives that begin with the same comment identifier. For example:

--This is a comment.  What follows is a directive with a parameter
--directive:param

A directive will always be in that format - two dash characters followed by a command, a colon, and a single parameter.

I would like to have the lexer ignore an actual comment (send it to the hidden channel), but tokenize the directives. I have the following lexer rules:

DCOMMAND    : DATABASE;
fragment DATABASE   : D A T A B A S E;
fragment COMMENTSTART   : '--';
LINE_COMMENT    : COMMENTSTART ~(DCOMMAND|('\n'|'\r')*) {$channel=HIDDEN;};
fragment A  : ('a'|'A');
fragment B  : ('b'|'B');
fragment C  : ('c'|'C');
fragment D  : ('d'|'D');
....

There's only one directive for now: 'database'. The DCOMMAND token will eventually represent several keywords potentially. The problem is that my lexer is always shoving anything that starts with '--' into the hidden channel. How do I make the LINE_COMMENT token not match directives? Or will I have to move comment handling into the parser?

1
antlrantlr3lexer
Can a comment start with --This:? In other words, is -- followed by an identifier followed by : always reserved for directives? – Sam Harwell
The latter is true. -- This: and --This also: should NOT be directives, however... note the whitespace. – DuckPuppy

1 Answers

3
votes

AFAIK, there's no way to handle this in your lexer grammar without some manual code (which is IMHO better than promoting comments to the parser!).

What you could do is this:

  • match '--'
  • in a custom method, manually look ahead until the end of the line (EOL). Let this method return true when the '--' is part of a directive
    • if what you matched until the EOL looks to be a directive, do NOT match the characters and return true
    • if what you matched until the EOL isn't a directive, match the characters and return false
  • if your custom method returned false, it must be a comment and you can skip() it

A quick demo:

grammar T;

@lexer::members {

  private boolean directiveAhead() throws MismatchedTokenException {

    StringBuilder b = new StringBuilder();

    for(int ahead = 1; ; ahead++) {

      // Grab the next character from the input.
      int next = input.LA(ahead);

      // Check if we're at the EOL.
      if(next == -1 || next == '\r' || next == '\n') {
        break;
      }

      b.append((char)next);
    }

    if(b.toString().trim().matches("\\w+:\\w+")) {
      // Do NOT let the lexer consume all the characters, just return true!
      return true;
    }
    else {
      // Let the lexer consume all the characters!
      this.match(b.toString());
      return false;
    }
  }
}

parse
 : directive EOF
 ;

directive
 : DIRECTIVE_START IDENTIFIER COL IDENTIFIER 
 ;

IDENTIFIER
 : ('a'..'z' | 'A'..'Z')+
 ;

DIRECTIVE_START
 : '--' { if(!directiveAhead()) skip(); }
 ;

COL
 : ':'
 ;

SPACES
 : (' ' | '\t' | '\r' | '\n')+ {skip();}
 ;