How can I import an ANTLR lexer grammar into another grammar using Gradle 2.10?

Question

I've been learning about ANTLR 4 with Terence Parr's The Definitive ANTLR 4 Reference, which I've been following so far using Gradle 2.10 and its built-in ANTLR plugin. However I'm having some trouble getting some code which I adapted from Chapter 4, pp. 38-41 to work properly with my Gradle build script. (The reason I'm using Gradle, rather than ANTLR directly, is because I want to eventually integrate ANTLR into a Java web application which I'm making for my dissertation, and I'd strongly prefer to use a build tool for this so I can automate the ANTLR-to-Java code generation process and easily manage my dependencies.)

I've created two ANTLR 4 grammars (which I've pasted towards the end of this question): src/main/antlr/org/jbduncan/Expr.g4 (a standard grammar) and src/main/antlr/org/jbduncan/CommonLexerRules.g4 (a lexer grammar), where Expr.g4 depends on CommonLexerRules.g4 via an import CommonLexerRules statement. But when I try to run gradlew generateGrammarSource at the command-line with my build.gradle (also pasted towards the end of this question along with my gradle.properties), I get the following error:

12:45:21: Executing external task 'generateGrammarSource'...
:generateGrammarSource
error(110): org\jbduncan\Expr.g4:3:7: can't find or load grammar CommonLexerRules
warning(125): org\jbduncan\Expr.g4:12:11: implicit definition of token NEWLINE in parser
warning(125): org\jbduncan\Expr.g4:13:6: implicit definition of token ID in parser
warning(125): org\jbduncan\Expr.g4:19:6: implicit definition of token INT in parser
:generateGrammarSource FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':generateGrammarSource'.
> There was 1 error during grammar generation

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 3.166 secs
There was 1 error during grammar generation
12:45:24: External task execution finished 'generateGrammarSource'.

Looking at this error message, it seems to me that Gradle's ANTLR plugin is able to find Expr.g4 but is somehow not able to find CommonLexerRules.g4.

I've made an attempt to resolve this error using a couple of external Gradle plugins (https://github.com/melix/antlr4-gradle-plugin and https://github.com/Abnaxos/gradle-antlr4-plugin) instead of the built-in one, but when I tried each of them, they introduced their own problems which I wasn't able to resolve.

Using the ANTLR 4.5.2 jar downloaded straight from the ANTLR website does let me to compile my two grammars without problems, but as I discussed earlier this is a really undesirable option for me, as I'd have to compile my grammars manually whereas I believe Gradle could do it automatically for me.

My question is this: How can I resolve the error above and get Gradle to import my lexer grammar into my other grammar?

---

build.gradle

apply plugin: 'java'
apply plugin: 'application'
apply plugin: 'antlr'

sourceCompatibility = 1.8
targetCompatibility = 1.8

[compileJava, compileTestJava]*.options*.encoding = 'UTF-8'

group = 'org.jbduncan'
version = '1.0-SNAPSHOT'
mainClassName = 'org.jbduncan.Application'

repositories {
    jcenter()
}

dependencies {
    antlr "org.antlr:antlr4:$antlrVersion"
    compile "com.google.guava:guava:$guavaVersion"
    testCompile "com.google.guava:guava-testlib:$guavaVersion"
    testCompile "com.google.truth:truth:$truthVersion"
    testCompile "junit:junit:$junitVersion"
}

// Send all generated source code to a directory other than build/, to workaround an issue in
// IntelliJ IDEA where it fails to recognise source files in build/.
def generatedJavaSourcesDir = 'src/generated/java'

generateGrammarSource {
    arguments += ['-visitor']
    outputDirectory = file(generatedJavaSourcesDir)
}

sourceSets {
    main {
        java {
            srcDir generatedJavaSourcesDir
        }
    }
}

clean {
    delete generatedJavaSourcesDir
}

task wrapper(type: Wrapper) {
    distributionUrl = "http://services.gradle.org/distributions/gradle-$gradleVersion-bin.zip"
}

gradle.properties

gradleVersion=2.10

# Dependency versions
antlrVersion=4.5
guavaVersion=19.0
junitVersion=4.12
truthVersion=0.28

src/main/antlr/org/jbduncan/Expr.g4

grammar Expr;

import CommonLexerRules; // includes all rules from CommonLexerRules.g4

@header {
    package org.jbduncan;
}

/** The start rule; begin parsing here. */
prog: stat+;

stat: expr NEWLINE # printExpr
    | ID '=' expr NEWLINE # assign
    | NEWLINE # blank
    ;

expr: expr op=('*'|'/') expr # MulDiv
    | expr op=('+'|'-') expr # AddSub
    | INT # int
    | ID # id
    | '(' expr ')' # parens
    ;

MUL: '*'; // assigns token name to '*' used above in grammar
DIV: '/';
ADD: '+';
SUB: '-';

src/main/antlr/org/jbduncan/CommonLexerRules.g4

lexer grammar CommonLexerRules; // note "lexer grammar"

ID: [a-zA-Z]+ ; // match identifiers
INT: [0-9]+ ; // match integers
NEWLINE: '\r'? '\n' ; // return newlines to parser (is end-statement signal)
WS: [ \t] -> skip ; // toss out whitespace

src/main/java/org/jbduncan/Application.java

package org.jbduncan;

import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTree;

import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;

public final class Application {
  public static void main(String[] args) throws IOException {
    String inputFile = (args.length > 0) ? args[0] : null;

    InputStream inputStream = (inputFile == null) ? System.in : new FileInputStream(inputFile);
    ANTLRInputStream input = new ANTLRInputStream(inputStream);
    ExprLexer lexer = new ExprLexer(input);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    ExprParser parser = new ExprParser(tokens);
    ParseTree tree = parser.prog(); // parse; start at prog

    System.out.println(tree.toStringTree(parser)); // print tree as text
  }
}

Answer 1

I managed to find an answer to my question after posting it on the Gradle forum:

generateGrammarSource {
  arguments << "-lib" << "src/main/antlr/org/jbduncan" 
}

where "src/main/antlr/org/jbduncan" is a path with your antlr .g4 library.