How can I tell Gradle to use my testng.xml file for Test Classes and Ordering?

Question

I am attempting to have Gradle execute some tests defined using a testng.xml file. My testng.xml file looks like this:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="mySuite" verbose="1">

  <listeners>
    <listener class-name="mypackage.MyListener" />
    <listener class-name="mypackage.TestOrderer" />
  </listeners>

  <test name="Tests">
    <classes>
      <class name="mytestpackage.CrazyTest1"/>
      <class name="mytestpackage.CrazyTest2"/>
      <class name="mytestpackage.CrazyTest3"/>
    </classes>
  </test>
</suite>

So why do I need this? I want to ensure that my tests are organized in a way that's defined by annotations similar to that which was listed here. As you might guess, TestOrderer is an IMethodInterceptor.

Here's the problem, Gradle seems to be taking over my testng.xml file and scraping the test directory to find the tests it wants to execute. Even if I disable this, it fails to execute the methods appropriately. Here's what I think should work, but just, doesn't.

test {
  useTestNG()
  options.suites("src/test/resources/crazyTestNG.xml") 
  # Adding 
  # scanForTestClasses = false 
  # causes zero tests to be executed, since the class names don't end in Test
}

It seems like it should be a no-brainer...fork the TestNG process and let it run, but I can't figure out how to tell Gradle to get out of the way and just execute my tests.

Answer 1

Here's how you can configure a test suite (xml) to be executed in a Gradle task:

apply plugin: 'java'

repositories {
    mavenCentral()
}
dependencies {
    // add the dependencies as needed
    testCompile group: 'org.testng', name: 'testng', version:'6.8.8'
    testCompile fileTree('lib')
}
test {
    useTestNG() {
        // runlist to executed. path is relative to current folder
        suites 'src/test/resources/runlist/my_test.xml'
    }
}

Answer 2

I hated the TestNG support in gradle... Found it to be most un-flexable compared to using raw TestNG. And I was tired of fiddling with gradle. My solution.. Run TestNG directly using a Gradle task

task runTests(type: JavaExec, dependsOn: 'classes') {
    main = 'org.testng.TestNG'
    classpath = files("./src/test/resources",
                      project.sourceSets.main.compileClasspath,
                      project.sourceSets.test.compileClasspath,
                      project.sourceSets.main.runtimeClasspath,
                      project.sourceSets.test.runtimeClasspath)
    args = ["-parallel",  "methods", "-threadcount", "1", "-d", "./build/test-output", "./src/test/resources/test.xml"]
}

Which I run from the command line:

gradle runTests

Answer 3

The Gradle TestNG runner assumes that if no test classes are specified, either by scanning for them, or pattern matching on the name, then it should skip test execution entirely.

Instead, it should consider whether a suite xml has been provided or not as well. Could you add a jira issue for this problem?

One possible work around is to use options.listener to specify the listeners, and not use a suite xml file at all:

test {
   options.listeners << 'mypackage.MyListener'
   options.listeners << 'mypackage.TestOrderer'
}

This way you don't need to specify the test classes, and you can just let the scanning find them for you.

Answer 4

This method doesn't use your testng.xml file but you could also emulate testNG test groups and order by creating JUnit test groups as Gradle tasks and then order them manually by ordering the task execution when you execute your build:

// in build.gradle
task testGroupOne(type: Test) {
   //include '**/*SuiteOne.*'
   include '**/SuiteOne.class'
   testReportDir = file("${reportsDir}/SuiteOne")
   testResultsDir = file("${buildDir}/test-results/SuiteOne")
}

task testGroupTwo(type: Test) {
   //include '**/*SuiteTwo.*'
   include '**/SuiteTwo.class'
   testReportDir = file("${reportsDir}/SuiteTwo")
   testResultsDir = file("${buildDir}/test-results/SuiteTwo")
}

Then, run your build like: gradle testGroupTwo testGroupOne

Answer 5

As another answer points out: the solution is to use the suites command. Though I prefer parameterizing the argument for that command so that from the command line I can choose any TestNG suite I want to run.

test {
    
    // Detect if suite param was passed in
    def runSuite = project.hasProperty("suite")
    
    useTestNG() {
        if (runSuite) {
            // If parameter was passed in, use it in the 'suites' command
            def suiteToRun = project.getProperty("suite")
            suites "src/test/resources/"+suiteToRun
        } else {
            // Handle non-command line executions e.g. running tests from IDE
            parallel 'methods'
            threadCount 2
        }
    }
}

Then from the command line I can run something like:

gradle test -Psuite=mysuite.xml

I prefer this to defining a bunch of custom Gradle tasks since that approach results in a messy build.gradle file and is slightly less flexible to new suites being added.