Maven failsafe: followed guide, but does not run on “verify”

Question

I am trying to make the Maven failsafe plugin run my integration tests, but even though I have followed the guide at https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html it doesn't work.

My maven project (MvnTest) looks like this (source files are below):

pom.xml
-src
  -main
    -java
    -resources
 -test
  -java
    SomeIT.java
    SomeTest.java

mvn test correctly runs only the tests in SomeTest.java using the sunfire plugin. Output:

...
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MvnTest ---
[INFO] Surefire report directory: ...
...
Running SomeTest
Tests run: 1, Failures: 1

However, the mvn verify command outputs exactly the same thing as mvn test which is wrong in two ways: 1) it calls sunfire, and 2) it runs the tests in SomeTest.java but should only run those in SomeIT.java (https://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html)

I believe I have followed the guide on using failsafe correctly, but obviously I must have missed something. What did I do wrong?

SOURCE FILES

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>MvnTest</groupId>
    <artifactId>MvnTest</artifactId>
    <version>1.0-SNAPSHOT</version>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.16</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.11</version>
        </dependency>
    </dependencies>
</project>

SomeIT.java:

import org.junit.Test;
import static org.junit.Assert.assertTrue;

public class SomeIT {
    @Test
    public void test2() {
        assertTrue(4 == 3);
    }
}

SomeTest.java:

import org.junit.Test;
import static org.junit.Assert.assertTrue;

public class SomeTest {
    @Test
    public void test1() {
        assertTrue(2 == 3);
    }
}

As you mentioned Tests run: 1, Failures: 1 this means that you usually should see BUILD FAILURE at the end which with an appropriate message about the failed unit tests. Your unit test will fail and based on the lifecyle maven says it does not make sense to continue with the integration tests if unit tests have faile. So you have to fix your unit test first which means mvn clean package should produce a BUILD SUCCESS message. Afterwards you can try to doe mvn verify. — khmarbaise
Thanks, but this does not work. mvn clean package produces BUILD FAILURE; it seems it runs the tests too. However I do not understand your comment about not running integration-tests if unit tests fails. Isn't the whole point of failsafe to separate the integration-tests from the unit-tests so that you can run them separately without affecting each other (besides compilation errors maybe)? — johnrl
mvn clean package will run unit tests but not the integration test if you have followed the naming conventions which looks like so. May be you can show the full pom file or put that on github or something similar to take a deeped look. If you run mvn verify the whole life cycle will run through which are stations compile, test, package, integration-test etc. — khmarbaise
khmarbaise is right. What's happening here is that you're not getting as far as running the integration tests using Failsafe - because your unit tests are failing first and thus stopping the build. Integration tests depend upon the unit tests passing because they execute the whole code stack. If any pieces of the stack had failed their unit test, it wouldn't make sense to try to integration test the whole stack when parts of it were broken. Once the unit tests pass, then Failsafe will kick-in. — Will Keeling
I understand now - thanks!. However, it is not true that it doesn't make sense to execute "integration tests" when unit tests fail. In my case I actually don't have "integration" tests, instead I have "acceptance" tests (ATs) written in cucumber as part of the BDD cycle. In my BDD the cycle is (AT -> (UT -> pass)*), so I write an AT, then I write several UTs to make it pass. In this case it is nice to be able to run the ATs at ANY point regardless of the state of the UTs. Also, why should I be bothered with UT results in the console, when I only want to see the results of the ATs? — johnrl

Eddú Meléndez Eddú Meléndez · Accepted Answer · 2014-01-15T16:34:43

Add includes section in your pom.xml for maven-failsafe-plugin

Looks like:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.16</version>
            <configuration>
                <includes>
                    <include>**/*IT.java</include>
                </includes>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Maven failsafe: followed guide, but does not run on “verify”

1 Answers