How to run parallel Cucumber Spring Boot integration tests with Maven (surefire or failsafe)?

Question

Hello folks,
I'm developing cucumber tests since january of this year and i'm now facing a problem running cucumber integration tests in parallel with maven (surefire as well as failsafe) in a Spring Boot environment.

In a nutshell:
I want to run cucumber integration tests in parallel with a spring boot application running. I use maven with either the Surefire or Failsafe plugin active. Everything works fine when i comment out every Spring Boot component in the code. Then i have as many threads as feature-files and they run parallel. But when I add the Spring Boot components I need, the tests run serial.

My dependencies in the pom.xml:

<dependencies>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java8</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-junit</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-spring</artifactId>
        <version>${cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>1.18.10</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
        <version>${spring.boot.version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
        <version>${spring.boot.version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <version>${spring.boot.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.30</version>
    </dependency>
    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <version>1.4.200</version>
        <scope>test</scope>
    </dependency>
</dependencies>

My plugins in the pom.xml:

<plugins>
    <!-- Maven Compiler -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
            <source>8</source>
            <target>8</target>
        </configuration>
    </plugin>
    <!-- Option 1 to run cucumber tests in parallel: Surefire -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.22.0</version>
        <configuration>
            <parallel>methods</parallel>
            <useUnlimitedThreads>true</useUnlimitedThreads>
        </configuration>
    </plugin>
    <!-- Option 2 to run cucumber tests in parallel: Failsafe -->
    <!--<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>3.0.0-M4</version>
        <executions>
            <execution>
                <goals>
                    <goal>integration-test</goal>
                    <goal>verify</goal>
                </goals>
                <configuration>
                    <parallel>methods</parallel>
                    <threadCount>5</threadCount>
                    <useUnlimitedThreads>false</useUnlimitedThreads>
                    &lt;!&ndash; important for the right execution &ndash;&gt;
                    <perCoreThreadCount>true</perCoreThreadCount>
                </configuration>
            </execution>
        </executions>
    </plugin>-->
</plugins>

My Java classes:

RunCucumberIT.java --> execution with failsafe
RunCucumberTest.java --> same code as RunCucumberIT.java, other name for execution with surefire
StepDefinitions.java
TestApplication.java
CucumberSpringContextConfiguration.java

The Java code:

RunCucumberIT.java and RunCucumberTest.java

package parallel;
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;
/**
 * Class to run the cucumber tests with Surefire.
 */
@RunWith(Cucumber.class)
@CucumberOptions(plugin = { }, strict = true, snippets = CucumberOptions.SnippetType.CAMELCASE,
        features = "src/test/java/parallel", stepNotifications = true)
public class RunCucumberTest {
}

StepDefinitions.java

package parallel;
import io.cucumber.java8.En;
import lombok.extern.slf4j.Slf4j;
@Slf4j
public class StepDefinitions implements En {
    public StepDefinitions() {
        Given("Step from {string} in {string} feature file", this::printThreads);
    }
    private void printThreads(final String scenario, final String file) throws InterruptedException {
        log.info(" Thread ID - {} - {} from {} feature file.", Thread.currentThread().getId(), scenario, file);
        Thread.sleep(5000L);
    }
}

TestApplication.java

package parallel;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class TestApplication {
    public static void main(final String[] args) {
        SpringApplication.run(TestApplication.class, args);
    }
}

CucumberSpringContextConfiguration.java

package parallel;
import io.cucumber.java.Before;
import io.cucumber.spring.CucumberContextConfiguration;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.test.context.SpringBootContextLoader;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ContextConfiguration;
/**
 * Class to use spring application context while running cucumber.
 */
@Slf4j
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ContextConfiguration(classes = TestApplication.class, loader = SpringBootContextLoader.class)
@CucumberContextConfiguration
public class CucumberSpringContextConfiguration {
    /**
     * Need this method so the cucumber will recognize this class as glue and load spring context configuration.
     */
    @Before
    public void setUp() {
        log.info("Spring Context initialized for executing cucumber tests.");
    }
}

My gherkin code:

scenario-outline.feature

Feature: Scenario Outlines feature file
  Scenario Outline: <scen_out_row_num>
    Given Step from '<scen_out_row_num>' in 'scenario-outlines' feature file
    Examples:
      | scen_out_row_num       |
      | Scenario Outline Row 1 |
      | Scenario Outline Row 2 |

scenarios-0.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One -
    Given Step from 'Scenario 1' in 'scenarios-0' feature file
  Scenario: Scenario Number Two -
    Given Step from 'Scenario 2' in 'scenarios-0' feature file

scenarios-1.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One a
    Given Step from 'Scenario 1' in 'scenarios-1' feature file
  Scenario: Scenario Number Two a
    Given Step from 'Scenario 2' in 'scenarios-1' feature file

scenarios-2.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One b
    Given Step from 'Scenario 1' in 'scenarios-2' feature file
  Scenario: Scenario Number Two b
    Given Step from 'Scenario 2' in 'scenarios-2' feature file

scenarios-3.feature

Feature: Scenarios feature file
  Scenario: Scenario Number One c
    Given Step from 'Scenario 1' in 'scenarios-3' feature file
  Scenario: Scenario Number Two c
    Given Step from 'Scenario 2' in 'scenarios-3' feature file

Expected behaviour:
One thread per feature-file and parallel execution of the integration tests. Some output like that (this is the output when i comment out all Spring Boot stuff):

[INFO] --- maven-failsafe-plugin:3.0.0-M4:integration-test (default) @ cucumber-test ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running Scenario Number One b
[pool-1-thread-4] INFO parallel.StepDefinitions -  Thread ID - 17 - Scenario 1 from scenarios-2 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.338 s - in Scenario Number One b
[INFO] Running Scenario Number One a
[pool-1-thread-3] INFO parallel.StepDefinitions -  Thread ID - 16 - Scenario 1 from scenarios-1 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.353 s - in Scenario Number One a
[INFO] Running Scenario Outline Row 1
[pool-1-thread-1] INFO parallel.StepDefinitions -  Thread ID - 14 - Scenario Outline Row 1 from scenario-outlines feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.363 s - in Scenario Outline Row 1
[INFO] Running Scenario Number One c
[pool-1-thread-5] INFO parallel.StepDefinitions -  Thread ID - 18 - Scenario 1 from scenarios-3 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.371 s - in Scenario Number One c
[INFO] Running Scenario Number One -
[pool-1-thread-2] INFO parallel.StepDefinitions -  Thread ID - 15 - Scenario 1 from scenarios-0 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.378 s - in Scenario Number One -
[INFO] Running Scenario Outline Row 2
[pool-1-thread-1] INFO parallel.StepDefinitions -  Thread ID - 14 - Scenario Outline Row 2 from scenario-outlines feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.003 s - in Scenario Outline Row 2
[INFO] Running Scenario Number Two a
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.011 s - in Scenario Number Two a
[INFO] Running Scenario Number Two c
[pool-1-thread-3] INFO parallel.StepDefinitions -  Thread ID - 16 - Scenario 2 from scenarios-1 feature file.
[pool-1-thread-5] INFO parallel.StepDefinitions -  Thread ID - 18 - Scenario 2 from scenarios-3 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.017 s - in Scenario Number Two c
[INFO] Running Scenario Number Two b
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.027 s - in Scenario Number Two b
[INFO] Running Scenario Number Two -
[pool-1-thread-4] INFO parallel.StepDefinitions -  Thread ID - 17 - Scenario 2 from scenarios-2 feature file.
[pool-1-thread-2] INFO parallel.StepDefinitions -  Thread ID - 15 - Scenario 2 from scenarios-0 feature file.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.043 s - in Scenario Number Two -


10
 Scenarios (
10 passed
)


10
 Steps (
10 passed
)


0m
10,486s

Actual behaviour:
Just one thread is used and all tests run serially. Output:

[INFO] --- maven-failsafe-plugin:3.0.0-M4:integration-test (default) @ cucumber-test ---
...    
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.4.RELEASE)
...
Thread ID - 1 - Scenario Outline Row 1 from scenario-outlines feature file.
2020-06-10 12:04:02.519  INFO 13028 --- [           main] o.s.t.c.support.AbstractContextLoader    : Could not detect default resource locations for test class [parallel.CucumberSpringContextConfiguration]: no resource found for suffixes {-context.xml, Context.groovy}.
...
2020-06-10 12:04:02.544  INFO 13028 --- [           main] parallel.StepDefinitions                 : 
... 
Thread ID - 1 - Scenario Outline Row 2 from scenario-outlines feature file.
2020-06-10 12:04:07.550  INFO 13028 --- [           main] o.s.t.c.support.AbstractContextLoader    : Could not detect default resource locations for test class [parallel.CucumberSpringContextConfiguration]: no resource found for suffixes {-context.xml, Context.groovy}.
Thread ID - 1 - Scenario 1 from scenarios-0 feature file.
...
Thread ID - 1 - Scenario 2 from scenarios-0 feature file.
...
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

I hope this problem is understandable and someone can help me!

Thanks in advance,
Maximotus

You may want to redo everything with stepNotifications = false . The stepNotifications confuse surefire/failsafe in lots of different ways. — M.P. Korstanje
When using @CucumberContextConfiguration you don't need the dummy step definition any more. — M.P. Korstanje
@M.P.Korstanje Thanks for that hint, but it results in the same actual output (for both failsafe and surefire). I think the problem is Spring Boot that somehow obstructs multi-threading with its context or something. — maximotus
@M.P.Korstanje Yeah you are right. I removed it. But still the same problem. Just the lines with "INFO 13028 --- [ main] p.CucumberSpringContextConfiguration : Spring Context initialized for executing cucumber tests." are missing in the output, as expected. — maximotus
Beats me. I wrote tests to confirm it works a while ago. You'll have to deep dive. — M.P. Korstanje

Melissa Vargas Melissa Vargas · Accepted Answer · 2020-10-26T11:17:14

For running cucumber test in parallel, you have to add in the pom.xml the exclusions for juni-jupiter and junit-vintage. This is how it worked for me:

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.jupiter</groupId>
                    <artifactId>junit-jupiter</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

How to run parallel Cucumber Spring Boot integration tests with Maven (surefire or failsafe)?

1 Answers