How do I run my cucumber-jvm features in parallel? (with maven using cucumber-jvm-parallel-plugin)

4
votes

I am trying to incorporate a maven plugin cucumber-jvm-parallel-plugin into my Cucumber-JVM code and am running into some problems... I think I have configured my pom.xml properly, but my cucumber features are still getting ran one by one instead of in parallel.

How do I configure my pom.xml to make my cucumber features run in parallel? I have some feature files that block execution and I don't want to have to wait for them to finish in order to start testing the other features.

I followed two tutorials, and can't find where I am going wrong, or if this is expected:

Structure

Here's my program's structure:

├── features/
│   └── api/
│       ├── 006-Email.feature
│       └── 999-Login.feature
├── results/
│   └── api-json.json
└── src/
    └── java/
        ├── pom.xml
        ├── src/
        │   └── test/
        │       └── java/
        │           └── com/
        │               └── mycompany/
        │                   └── commonapps/
        │                       └── queuemanager/
        │                           ├── Globals.java
        │                           ├── GmailHelper.java
        │                           ├── HttpDeleteWithBody.java
        │                           ├── JsonHelper.java
        │                           ├── RequestHelper.java
        │                           ├── RunApiTest.java
        │                           ├── ThreadedSteps.java
        │                           └── UserHelper.java
        └── steps/
            └── com/
                └── mycompany/
                    └── commonapps/
                        └── queuemanager/
                            ├── GivenSteps.java
                            ├── Setup.java
                            ├── ThenSteps.java
                            └── WhenSteps.java

Maven Pom Logic

The way I configured my pom is as follows:

  1. I copy ../../features/api/ to my build path using Maven Resources Plugin 

    <build>
  <resources>
     <resource>
        <directory>../../features/api/</directory> 
     </resource>
  </resources>
</build>

  2. I tell cucumber-jvm-parallel-plugin that my features are now located at target/classes/active/, because I think ../../features/api/ was copied to target/, so target/active/ should contain my .feature files.. I think.

     <plugin>
      <groupId>com.github.temyers</groupId>
      <artifactId>cucumber-jvm-parallel-plugin</artifactId>
      ...
      <configuration>
          ...
          <featuresDirectory>${project.build.directory}/classes/active/</featuresDirectory>
          ...
      </configuration>
 </plugin>

  3. Then, after I run maven test, all .feature files are then compiled into target/test-classes/Parallel**IT.class. One per feature file.

  4. So I tell maven-surefire-plugin to run all <include>**/Parallel*IT.class</include> files in parallel, which are the feature files created by cucumber-jvm-parallel-plugin.

     <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <executions>
       <execution>
          <id>acceptance-test</id>
          <phase>integration-test</phase>
          <goals>
             <goal>test</goal>
          </goals>
          <configuration>
            <forkCount>5</forkCount>
            <reuseForks>true</reuseForks>
            <includes>
              <include>**/Parallel*IT.class</include>
            </includes>
          </configuration>
       </execution>
    </executions>
 </plugin>

Full pom.xml

And my pom.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <groupId>com.mycompany.commonapps</groupId>
   <artifactId>e2e-server</artifactId>
   <version>0.0.1-SNAPSHOT</version>
   <dependencies>
     ...
   </dependencies>
   <build>
<!-- COPY RESOURCES TO TARGET/ -->
      <resources>
    <!-- copies features to target: "${project.build.directory}/classes/active/" -->
         <resource>
            <directory>../../features/api/</directory> 
         </resource>
      </resources>
<!-- RUN EACH CUCUMBER FEATURE FILE AS A FORK -->
      <plugins>
    <plugin>
      <groupId>com.github.temyers</groupId>
      <artifactId>cucumber-jvm-parallel-plugin</artifactId>
      <version>1.2.1</version>
      <executions>
        <execution>
          <id>generateRunners</id>
          <phase>generate-test-sources</phase>
          <goals>
        <goal>generateRunners</goal>
          </goals>
          <configuration>
          <glue>com.mycompany.commonapps.queuemanager</glue>
           <outputDirectory>${project.build.directory}/generated-test-sources</outputDirectory>
           <featuresDirectory>${project.build.directory}/classes/active/</featuresDirectory>
           <cucumberOutputDir>${project.build.directory}</cucumberOutputDir>
           <format>json</format>
           <!--<strict>true</strict>
           <monochrome>true</monochrome> 
           <useTestNG>false</useTestNG>
           <namingScheme>simple</namingScheme>-->
           <parallelScheme>FEATURE</parallelScheme>
          </configuration>
        </execution>
      </executions>
    </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
<!-- RUN ALL CLASSES CREATED BY cucumber-jvm-parallel-plugin IN PARALLEL -->
        <executions>
         <execution>
             <id>acceptance-test</id>
             <phase>integration-test</phase>
             <goals>
                 <goal>test</goal>
             </goals>
             <configuration>
                 <forkCount>5</forkCount>
                 <reuseForks>true</reuseForks>
                 <includes>
                    <include>**/Parallel*IT.class</include>
                 </includes>
             </configuration>
        </execution>
        </executions>
         </plugin>
<!-- ADD ../../STEPS/ TO THE SOURCE -->
         <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>1.7</version>
            <executions>
               <execution>
                  <id>add-source</id>
                  <phase>generate-sources</phase>
                  <goals>
                     <goal>add-source</goal>
                  </goals>
                  <configuration>
                     <sources>
                        <source>steps/</source>
                        <source>src/test/java/</source>
                     </sources>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
</project>

Cucumber Runner

And my cucumber running class looks like this:

@RunWith(Cucumber.class)
@CucumberOptions(plugin={"pretty","json:../../results/api-json.json"}, glue={"com.mycompany.commonapps.queuemanager"}, features={"../../features/api/active"})
public class RunApiTest {
}

Output

When I run mvn test, I get the following:

...
[INFO] ------------------------------------------------------------------------
[INFO] Building e2e  Server Tests 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The artifact org.apache.commons:commons-io:jar:1.3.2 has been relocated to commons-io:commons-io:jar:1.3.2
[INFO] 
[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @ e2e--server ---
[INFO] Source directory: /media/ifc-dev-1/ESD-ISO/e2e-/src/java/steps added.
[INFO] Source directory: /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/java added.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ e2e--server ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 7 resources
[INFO] Copying 2 resources
[INFO] Copying 7 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ e2e--server ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 12 source files to /media/ifc-dev-1/ESD-ISO/e2e-/src/java/target/classes
[INFO] /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/java/com/mycompany/commonapps/queuemanager/GmailHelper.java: /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/java/com/mycompany/commonapps/queuemanager/GmailHelper.java uses unchecked or unsafe operations.
[INFO] /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/java/com/mycompany/commonapps/queuemanager/GmailHelper.java: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- cucumber-jvm-parallel-plugin:1.2.1:generateRunners (generateRunners) @ e2e--server ---
[INFO] Adding /media/ifc-dev-1/ESD-ISO/e2e-/src/java/target/generated-test-sources to test-compile source root
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ e2e--server ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.2:testCompile (default-testCompile) @ e2e--server ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 10 source files to /media/ifc-dev-1/ESD-ISO/e2e-/src/java/target/test-classes
[INFO] /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/java/com/mycompany/commonapps/queuemanager/GmailHelper.java: /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/java/com/mycompany/commonapps/queuemanager/GmailHelper.java uses unchecked or unsafe operations.
[INFO] /media/ifc-dev-1/ESD-ISO/e2e-/src/java/src/test/java/com/mycompany/commonapps/queuemanager/GmailHelper.java: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ e2e--server ---
[INFO] Surefire report directory: /media/ifc-dev-1/ESD-ISO/e2e-/src/java/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.mycompany.commonapps.queuemanager.RunApiTest
....
*Tests run one by one properly*
...
Tests run: 31, Failures: 2, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:40 min
[INFO] Finished at: 2016-11-10T15:49:11-08:00
[INFO] Final Memory: 20M/261M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project e2e--server: There are test failures.
[ERROR] 
[ERROR] Please refer to /media/ifc-dev-1/ESD-ISO/e2e-/src/java/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Does anyone see what I did wrong?

More Debugging Info

  • cucumber-jvm-parallel-plugin's outputDirectory is getting created correctly and contains a bunch of Parallel**IT.class files
  • cucumber-jvm-parallel-plugin's featuresDirectory is getting accessed correctly.
  • cucumber-jvm-parallel-plugin's cucumberOutputDir is NOT getting created at all after a build.
1
mavencucumbercucumber-jvmcucumber-junitcucumber-java
I would suggest that you run Maven using the -X and perhaps -e options to get more information from your build. Do a mvn -h to get all options available. It is also worth noting that Cucumber is not thread safe and does not support parallelism. You scenarios may or may not work when they are executed in parallel. – Thomas Sundberg
Thanks @ThomasSundberg, I'll try that – Katie
i would suggest to take a look QAF Gherkin Scenario Factory – user861594
Please clarify what do you want to use cucumber-jvm-parallel-plugin first place ? Why don't you just use maven-failsafe-plugin to run each of the .feature files in parallel ? – vikramvi

1 Answers

1
votes

I finally got it to work! Here's what I had to do to make my tests run in parallel:

  1. I move some fields in the maven-surefire-plugin execution tag to the configuration tag. I moved this section:

         <forkCount>5</forkCount>
     <reuseForks>true</reuseForks>
     <includes>
        <include>**/*IT.class</include>
     </includes>
    • So that my maven-surefire-plugin now looks like this:

        <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.19.1</version>

      <configuration>
         <forkCount>5</forkCount>
         <reuseForks>true</reuseForks>
         <includes>
             <include>**/*IT.class</include>
         </includes>
      </configuration>

    </plugin>

  2. Since cucumber-jvm-parallel-plugin is now running the show, I had to remove my cucumber runner class. I commented it out like this:

    /*
    @RunWith(Cucumber.class)
    @CucumberOptions(plugin={"pretty","json:../../results/api-json.json"}, glue={"com.mycompany.commonapps.queuemanager"}, features={"../../features/api/active"})
    */
    public class RunApiTest {
    }

And that's it. Since Parallel**IT.class where getting created, it was maven-surefire-plugin that was not working right.