Antlr 4 Lexer with multiple modes failing to tokenise correctly

Question

I'm trying to create a lexer with multiple modes using Antlr 4.7. My lexer currently is:

ACTIONONLY  : 'AO'; 

BELIEFS :   ':Initial Beliefs:' -> mode(INITIAL_BELIEFS);
NAME    :   ':name:';
WORD:   ('a'..'z'|'A'..'Z'|'0'..'9'|'_')+;

COMMENT : '/*' .*? '*/' -> skip ;
LINE_COMMENT : '//' ~[\n]* -> skip ;
NEWLINE:'\r'? '\n' -> skip  ;
WS  :   (' '|'\t') -> skip ;

mode INITIAL_BELIEFS;
GOAL_IB :   ':Initial Goal:' -> mode(GOALS);
IB_COMMENT : '/*' .*? '*/' -> skip ;
IB_LINE_COMMENT : '//' ~[\n]* -> skip ;
IB_NEWLINE:'\r'? '\n' -> skip  ;
IB_WS  :   (' '|'\t') -> skip ;
BELIEF_BLOCK: ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'('|')'|','|'.')+;

mode REASONING_RULES;
R1: 'a';
R2: 'b';

mode GOALS;
GL_COMMENT : '/*' .*? '*/' -> skip ;
GL_LINE_COMMENT : '//' ~[\n]* -> skip ;
GL_NEWLINE:'\r'? '\n' -> skip  ;
GL_WS  :   (' '|'\t') -> skip ;
GOAL_BLOCK: ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'('|')'|','|'.')+;

Note that there is no way, at present, to get into the REASONING_RULES mode (so this should not, as I understand it have any effect on the operation of the lexer). Obviously I do want to use this mode, but this is the minimal version of the lexer that seems to display the problem I'm having.

My parser is:

grammar ActionOnly;

options { tokenVocab = ActionOnlyLexer; }

// Mas involving ActionOnly Agents
mas  :  aoagents;

aoagents: ACTIONONLY (aoagent)+;

// Agent stuff
aoagent  : 
    (ACTIONONLY?) 
    NAME w=WORD  
    BELIEFS (bs=BELIEF_BLOCK )?
    GOAL_IB gs=GOAL_BLOCK;

and I'm trying to parse:

AO

:name: robot

:Initial Beliefs:

abelief

:Initial Goal:

at(4, 2)

This fails with the error

line 35:0 mismatched input 'at(4,' expecting GOAL_BLOCK

which I'm assuming is because it isn't tokenising correctly.

If I omit rule R2 in the REASONING_RULES mode then it parses correctly (in general I seem to be able to have one rule in REASONING_RULES and it will work, but more than one rule and it fails to match GOAL_BLOCK)

I'm really struggling to see what I'm doing wrong here, but this is the first time I've tried to use lexer modes with Antlr.

Bart Kiers Bart Kiers · Accepted Answer · 2017-08-16T20:53:34

I don't get that error when I try your grammars. I also tested with ANTLR 4.7.

Here's my test rig:

import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.Token;

public class Main {

    public static void main(String[] args) {

        String source = "AO\n" +
                "\n" +
                ":name: robot\n" +
                "\n" +
                ":Initial Beliefs:\n" +
                "\n" +
                "abelief\n" +
                "\n" +
                ":Initial Goal:\n" +
                "\n" +
                "at(4, 2)";

        ActionOnlyLexer lexer = new ActionOnlyLexer(CharStreams.fromString(source));
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        tokens.fill();

        System.out.println("[TOKENS]");

        for (Token t : tokens.getTokens()) {
            System.out.printf("  %-20s %s\n", ActionOnlyLexer.VOCABULARY.getSymbolicName(t.getType()), t.getText());
        }

        System.out.println("\n[PARSE-TREE]");

        ActionOnlyParser parser = new ActionOnlyParser(tokens);
        ParserRuleContext context = parser.mas();

        System.out.println("  "+context.toStringTree(parser));
    }
}

And this is printed to my console:

[TOKENS]
  ACTIONONLY           AO
  NAME                 :name:
  WORD                 robot
  BELIEFS              :Initial Beliefs:
  BELIEF_BLOCK         abelief
  GOAL_IB              :Initial Goal:
  GOAL_BLOCK           at(4,
  GOAL_BLOCK           2)
  EOF                  <EOF>

[PARSE-TREE]
  (mas (aoagents AO (aoagent :name: robot :Initial Beliefs: abelief :Initial Goal: at(4,)))

Perhaps you need to generate new lexer/parser classes?

PS. note that ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'('|')'|','|'.')+ can be written as [a-zA-Z0-9_(),.]+

Antlr 4 Lexer with multiple modes failing to tokenise correctly

1 Answers