Why failsafe plugin requires both integration-test and verify goals?

7
votes

I have the next pom.xml

<project>
   ...
     <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.6</version>
            <executions>
                <execution>
                    <id>integration-test</id>
                    <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                    </goals>
                    <configuration>
                        <argLine>${failsafeArgLine}</argLine>
                        <includes>
                            <include>**/dmg/*IT.java</include>
                        </includes>
                        <skipTests>${skipTests}</skipTests>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    ...
</project>

The problem is that when I'm taking off verify goal then the build is every time successful even if there was test failures.

And when I'm Taking off integration-test goal the integration tests simply do not run

Why failsafe plugin requires both integration-test and verify goals?

2
mavenintegration-testingmaven-failsafe-plugin
What are you using such an extreme old version of maven-failsafe-plugin. The current version is 2.17...Furthermore the includes is not necessary cause these are the defaults. – khmarbaise
This includes is necessary because i need to run only IT which is in dmg package and thank you... I didn't know about version)) – gokareless

2 Answers

11
votes

In Maven Failsafe plugin reference you can find simple answer why build is always successful 

failsafe:integration-test runs the integration tests of an application.
  failsafe:verify verifies that the integration tests of an application passed.

Without verify goal test results are not checked at all(but they are executed), so failsafe plugin requires integration-test goal to run tests, and verify to "verify" their results.

3
votes

In Maven there are two types of test runner plug-ins, Surefire and Failsafe and they both serve different purposes. (I mention Surefire here because I feel it helps explain and contrast different types of failure in builds.)

Surefire

Surefire is a plug-in designed for your pre-deploy tests, such as Unit and Component tests. Surefire is guaranteed to return a failure (and therefore break your build) if any of the tests fail. You want this behaviour in pre-deploy tests because you want to fail the build early if your tests fail, and as Unit and Component tests don't (or shouldn't) have any external dependencies it's safe to fail a build there and then.

Failsafe

Failsafe is a plug-in designed for post-deploy tests, such as Functional and Smoke tests. Failsafe, as the name implies handles failures safely, by always returning a success exit code (as you have experienced in your build). The reason behind this is for post-deploy tests you don't want test failures to immediately break the build because you may have spun up infrastructure, or seeded some test data into the system, which needs clean-up before the build is failed. Therefore Failsafe splits the failure checking into the failsafe:verify goal which happens in the verify phase.

If you run Maven without parameters (assuming the normal jar lifecycle) you'll notice a phase between integration-test and verify called, post-integration-test. This is the phase which was designed for the tear down of any infrastructure or test data you may have injected into the system to run the tests.

... pre-integration-test, integration-test, post-integration-test, verify, ...

The phase before integration-test, pre-integration-test is designed for the said seeding and infrastructure creation.

To sum it all up, with pre-deploy tests you want to fail the build as early as possible, and so surefire guarantees this by exiting with a failure.

Whereas with post-deploy tests there are often external dependencies which need clean-up before you fail the build. For example in Kubernetes you may want to issue a rollback of the deploy and so failsafe:integration always returns a build success to guarantee you can perform the clean-up before failing the build by checking the status with failsafe:verify.

(The currently accepted answer just quotes the manual and doesn't explain why there are two commands and there are differences which is why I feel this answer is needed.)