ANTLR4 : Island grammar, token matching / skipping

Question

If fighting an island grammar with antlr4, and while I can make it work, I still have doubts if this is the "proper" way.

I need to parse :

Some random text
{ }

@if(condition) {
   more random text
   @foobar
   @if (condition2) {
        random text {}
   }
}

The problem lies within the context : An "wild" {} isn't anything, but if it's a { } behind a language operator, the { } become meaningful. (read : It opens and closes a block)

In the above case, it would return the following, assuming that condition and condition2 are both true :

Some random text
{}
more random text
random text {}

I'm confused on which route to pick, any advice on the above ?

The original implementation seems to be matching braces :

{ }

@if (true) {
    { 
    foo 
    bar  
    } }

yields

{ }
{
foo
bar
}

while

{ }

@if (true) {
{ 
    foo 
    bar  
    }

yields a parse error.

Parser / lexer are at : github.com/igmar/rythm2/blob/master/src/main/antlr4/… github.com/igmar/rythm2/blob/master/src/main/antlr4/… I've not implemented the parts referring to the question, since I'm still thinking about the right approach. — Igmar Palsenberg
To make clear what I mean : The current parser fails at { } as an input. Logical : it returns CURLY_OPEN CURLY_CLOSE, which isn't allowed according to the grammar. I could put this in a mode, which isn't easy, since I need to be able to escape it. Basically, I want to say : Match } if you've seen a { first. — Igmar Palsenberg
The { and } in the input random text {} are rather ambiguous. How do you make a distinction between the } in random text {} and the } that close a block? If they occur alone on a single line (as the last 2 }s), should they then become CURLY_CLOSE? If you can explain it in plain English, I might be able to help :) — Bart Kiers
You seem to be reverse engineering an existing parser, but you're not entirely clear what the exact rules are. It's hard telling a computer/parser how to handle things you yourself aren't entirely sure of :). I'd start by looking at the source of the existing parser and/or ask the people who wrote it for clues. — Bart Kiers

Igmar Palsenberg Igmar Palsenberg · Accepted Answer · 2016-07-18T15:00:07

this can be solved with a context specific lexer. In this case, by keeping track of the condition / block openings, we can determine if this is template content, or an actual block opening / closing.

See p219 of the ANTLR4 definitive ANTLR4 reference.

ANTLR4 : Island grammar, token matching / skipping

1 Answers