Why is TestNG running these test classes multiple times?

Question

I'm running builds and tests on a CI server (Bamboo). I'm using TestNG to run my integration tests.

When I run my build and tests (using Maven), I see the following in the logs (this is an example, there are other classes affected too)

build   27-Dec-2013 19:06:35    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$18
build   27-Dec-2013 19:06:36    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:06:36    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.194 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$18
build   27-Dec-2013 19:06:36    Running c.v.p.i.c.ExerciseLifecycleForStaticAssetIntegrationTest$1
build   27-Dec-2013 19:06:36    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:06:36    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.188 sec - in c.v.p.i.c.ExerciseLifecycleForStaticAssetIntegrationTest$1
build   27-Dec-2013 19:06:36    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$25
build   27-Dec-2013 19:06:36    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:06:36    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.188 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$25
build   27-Dec-2013 19:06:37    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest
build   27-Dec-2013 19:06:37    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:07:57    Tests run: 21, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 80.748 sec <<< FAILURE! - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest
build   27-Dec-2013 19:07:57    testGetPostDescriptorForVersion(c.v.p.i.c.MultiChannelContentServiceIntegrationTest)  Time elapsed: 0.066 sec  <<< FAILURE!
build   27-Dec-2013 19:07:57    java.lang.AssertionError: expected:<1.0> but was:<null>
build   27-Dec-2013 19:07:57        at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
build   27-Dec-2013 19:07:57        at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:108)
build   27-Dec-2013 19:07:57        at c.v.p.i.c.MultiChannelContentServiceIntegrationTest.checkFieldsThatShouldBeTheSameForAllVersions(MultiChannelContentServiceIntegrationTest.java:891)
build   27-Dec-2013 19:07:57        at c.v.p.i.c.MultiChannelContentServiceIntegrationTest.testGetPostDescriptorForVersion(MultiChannelContentServiceIntegrationTest.java:832)
build   27-Dec-2013 19:07:57    
build   27-Dec-2013 19:07:58    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$24
build   27-Dec-2013 19:07:58    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:07:58    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$24
build   27-Dec-2013 19:07:58    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$15
build   27-Dec-2013 19:07:58    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:07:58    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.183 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$15
build   27-Dec-2013 19:07:58    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$11
build   27-Dec-2013 19:07:59    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:07:59    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.2 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$11
build   27-Dec-2013 19:07:59    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$2
build   27-Dec-2013 19:07:59    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:07:59    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.206 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$2
build   27-Dec-2013 19:07:59    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$22
build   27-Dec-2013 19:07:59    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:00    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.189 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$22
build   27-Dec-2013 19:08:00    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$7
build   27-Dec-2013 19:08:00    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:00    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.19 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$7
build   27-Dec-2013 19:08:00    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$13
build   27-Dec-2013 19:08:00    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:00    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.199 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$13
build   27-Dec-2013 19:08:00    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$16
build   27-Dec-2013 19:08:01    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:01    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.214 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$16
build   27-Dec-2013 19:08:01    Running c.v.p.i.c.ExerciseLifecycleIntegrationTestTemplate
build   27-Dec-2013 19:08:01    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:01    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.229 sec - in c.v.p.i.c.ExerciseLifecycleIntegrationTestTemplate
build   27-Dec-2013 19:08:01    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$14
build   27-Dec-2013 19:08:01    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:02    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.184 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$14
build   27-Dec-2013 19:08:02    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$28
build   27-Dec-2013 19:08:02    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:02    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$28
build   27-Dec-2013 19:08:02    Running c.v.p.i.c.MultiChannelContentServiceIntegrationTest$5
build   27-Dec-2013 19:08:02    Configuring TestNG with: TestNGMapConfigurator
build   27-Dec-2013 19:08:02    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.191 sec - in c.v.p.i.c.MultiChannelContentServiceIntegrationTest$5

What I'm confused about is the instances of c.v.p.i.c.MultiChannelContentServiceIntegrationTest$XX.

I haven't intentionally or explicitly requested that the classes by run multiple times, nor does it seem that they are being run multiple times (since they report Test runs: 0 for all but a single instance).

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.16</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-testng</artifactId>
            <version>2.16</version>
        </dependency>
    </dependencies>
    <executions>
        <execution>
            <id>integration-test</id>
            <goals>
                <goal>integration-test</goal>
            </goals>
            <configuration>
                <argLine>
                    -javaagent:"${user.home}/.m2/repository/org/springframework/spring-instrument/${org.springframework.version}/spring-instrument-${org.springframework.version}.jar"
                    -Djava.awt.headless=true -Dehcache.rest.port=9888
                </argLine>
                <includes>
                    <include>c/v/p/integration/**/*.java</include>
                </includes>
                <reuseForks>false</reuseForks>
                <forkCount>1</forkCount>
                <argLine>-Xms512m -Xmx3096m -XX:PermSize=384m -XX:MaxPermSize=512m</argLine>
            </configuration>
        </execution>
    </executions>
</plugin>

Can anyone point me towards what might be happening here. The phenomenon doesn't really lend itself to Googling. Most searches for "multiple test runs with TestNG" explain how to do it on purpose.

Harald Wellmann Harald Wellmann · Accepted Answer · 2013-12-29T19:57:18

Are there any anonymous inner classes in your test class? That's what MultiChannelContentServiceIntegrationTest$22 etc. seems to indicate.

Apparently, the pattern

<include>c/v/p/integration/**/*.java</include>

includes these anonymous classes, so Surefire tries to run them as test classes.

Try **/*Test.java or simply use the default.

Why is TestNG running these test classes multiple times?

1 Answers