Contextual tokens in ANTLR?

0
votes

I am trying to integrate a language into NetBeans using ANTLR, and in order to highlight syntax this involves generating tokens for every type of text I want highlighted.

Is there a way to create tokens for contextual words? That is, suppose I want to highlight all Strings that come after "function":

function foo

where "foo" gets highlighted.

how might I define the token rule:

FUNCTION_IDENTIFIER
    :    //match anything after "function"

so that the token matches what I want?

1
syntax-highlightingantlrtokentokenize

1 Answers

1
votes

You could do that by overriding the emit() method from the lexer and keep track of the last emitted token. Then inside your IDENTIFIER rule, you check if the last token was a FUNCTION, in which case you set a different $type for said token.

A demo:

grammar T;

tokens {
  FUNCTION_IDENTIFIER;
}

@lexer::members {

  private Token last = null;

  @Override
  public Token emit() {
    last = super.emit();
    return last;
  }
}

parse
  :  (t=. {System.out.printf("\%-20s -> '\%s'\n", tokenNames[$t.type], $t.text);})* EOF
  ;

FUNCTION
  :  'function'
  ;

IDENTIFIER
  :  ('a'..'z' | 'A'..'Z')+ 
     {
       if(last != null && last.getType() == FUNCTION) $type=FUNCTION_IDENTIFIER;
     }
  ;

SPACE
  :  ' ' {skip();}
  ;

And if you run this class:

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    TLexer lexer = new TLexer(new ANTLRStringStream("a function b c"));
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    TParser parser = new TParser(tokens);
    parser.parse();
  }
}

you will see:

bart@hades:~/Programming/ANTLR/Demos/T$ java -cp antlr-3.3.jar org.antlr.Tool T.g
bart@hades:~/Programming/ANTLR/Demos/T$ javac -cp antlr-3.3.jar *.java
bart@hades:~/Programming/ANTLR/Demos/T$ java -cp .:antlr-3.3.jar Main
IDENTIFIER           -> 'a'
FUNCTION             -> 'function'
FUNCTION_IDENTIFIER  -> 'b'
IDENTIFIER           -> 'c'

EDIT

Note that if you have tokens written to the HIDDEN channel, you'll need to change the contents of emit() slightly. Something like this (untested!):

@lexer::members {

  private Token last = null;

  @Override
  public Token emit() {
    Token temp = super.emit();
    if(temp.getType() != HIDDEN) {
      last = temp;
    }
    return temp;
  }
}

EDIT II

will this break other rules that I had? Suppose I had a rule that took all IDENTIFIER tokens and I added this contextual token. Would the rule I previously had now ignore all FUNCTION_IDENTIFIERS, causing me to have to explicitly catch both FUNCTION_IDENTIFIER and IDENTIFIER in that rule?

Yes, any rule referencing IDENTIFIER will not match a FUNCTION_IDENTIFIER token. If you want that, simply create a production (parser rule) like this:

identifier
  :  IDENTIFIER
  |  FUNCTION_IDENTIFIER
  ;

and replace all IDENTIFIER usages in parser rules by identifier instead.