Grab all blank lines by Antlr 4 lexer/parser

Question

Trying to grab all blank lines (only line numbers) by Antlr 4 lexer/parser for a given PHP file. The grammar I am using is available on GitHub Antlr grammar for PHP.

The Whitespace token defined as:

Whitespace:         [ \t\r\n]+ -> skip;

I changed this to:

Whitespace: (
             ' ' 
             | '\t' 
             | '\r' '\n' { newline(); } 
             | '\n'       { newline(); }
            );

But it's collecting almost all the lines, since every line is ending by "\n". Any expert advice can give me a lead.

The sample PHP to test:

<?php

//02-5002201-00001 5002201 - Machine hours test

	function test()
	{	
/*	Name:			Test.php

	Title:			Demo

	by:				XYZ
*/
		if (true && false)
		{		
			echo "aa";
		}

//TODO		
		echo <<<SEGDTA
		<link rel="stylesheet" type="text/css" href="ui.css"/>

		<script type="text/javascript" src="min.js"></script>
		SEGDTA;
	}

?>

Answer 1

Try something like this instead:

lexer grammar DemoLexer;

EmptyLine
 : {super.getCharPositionInLine() == 0}? [ \t]* '\r'? '\n'
 ;

Whitespace
 : [ \t\r\n] -> skip
 ;

Other
 : .
 ;

If I run the following test class:

import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.Token;

public class Main {

    public static void main(String[] args) {

        String source = "foo\n" +
                "\n" +
                "bar\n" +
                "    \n" +
                "   baz   \r\n" +
                " \t\t\n" +
                "\tend\n\n";

        DemoLexer lexer = new DemoLexer(CharStreams.fromString(source));

        for (Token t : lexer.getAllTokens()) {

            String name = lexer.getTokenNames()[t.getType()];
            String text = t.getText().replace("\r","\\r").replace("\n","\\n").replace("\t","\\t");

            System.out.printf("%-20s '%s'\n", name, text);
        }
    }
}

this will be printed:

Other                'f'
Other                'o'
Other                'o'
EmptyLine            '\n'
Other                'b'
Other                'a'
Other                'r'
EmptyLine            '    \n'
Other                'b'
Other                'a'
Other                'z'
EmptyLine            ' \t\t\n'
Other                'e'
Other                'n'
Other                'd'
EmptyLine            '\n'

See: http://www.antlr.org/api/Java/org/antlr/v4/runtime/TokenSource.html#getCharPositionInLine()

Answer 2

Is this the only processing you do with the PHP code? If so you could simply load the file line by line and count empty entries. No parser needed in such a case.

Update

Since you have that parser anyway you could use the token stream and walk over all tokens. Whenever you see a linebreak check the previous token and if that is also a line break (or this is the first token in the stream) you found an empty line. You can even keep your whitespace hidden, since the token stream will give you all tokens on all channels (unless you filtered it).

Counting empty lines is a semantic step anyway and the parser (which is doing the syntactic step) is not the right place for this.

Update 2

Here's code that should work (based on your attempt):

CommonTokenStream tokenStream = new CommonTokenStream(new AntlrPHPLexer(charStream));

tokenStream.fill(); // Load all tokens.
int counter = 0;
List<Token> tokens = tokenStream.getTokens();
for (int i = 0; i < tokens.size(); ++i) {
  if (tokens.get(i).getType() == AntlrPHPLexer.Linebreak) {
    if (i == 0 || (tokens.get(i - 1).getType() == AntlrPHPLexer.Linebreak))
      ++counter;
  }
}

You have to split your whitespaces into 2 rules:

Whitespace: ([ \t]+ | Linebreak) -> skip;
Linebreak: [\r\n];

Note that I did not use a loop for the Linebreak call in Whitespace, by intention.