Parallel Cucumber-JVM Selenium Grid Tests using Maven Surefire

Question

I've been going around in circles trying to figure this out, it seems almost impossible to use cucumber with maven parallel options against a Selenium Grid cluster.

My runner runs based on tags and should pick up about 5 or 6 feature files.

I'm using Cucumber for Java, with Maven surefire 2.19 and Junit.

So far my surefire config for the pom looks like this :-

<profile>
        <id>RunFeature</id>
        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>${surefire.version}</version>
                    <configuration>
                        <threadCount>5</threadCount>
                        <parallel>methods</parallel>
                        <systemProperties>
                            <property>
                                <name>environmentName</name>
                            </property>
                        </systemProperties>
                        <includes>
                            <include>**/RunFeature.java</include>
                        </includes>
                        <testFailureIgnore>true</testFailureIgnore>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

My runner looks like this :-

@RunWith(Cucumber.class)
@CucumberOptions(
        plugin = {"json:target/cucumber.json"},
        features = "classpath:",
        glue = "classpath:",
        dryRun = false,
        strict = true)

public class RunFeature {

}

And I'm running with with Maven:-

clean test -P RunFeature -D environmentName="DEV" -Dcucumber.options="--tags @runnable --tags @testa--tags @testb"

And this is the stacktrace :-

    java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
    at java.util.ArrayList$Itr.next(ArrayList.java:851)
    at cucumber.runtime.RuntimeOptions$1.invoke(RuntimeOptions.java:287)
    at com.sun.proxy.$Proxy17.feature(Unknown Source)
    at cucumber.runtime.junit.JUnitReporter.feature(JUnitReporter.java:165)
    at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:69)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:93)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:37)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:387)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Feature: Buyer guidelines on buyer detailscucumber.runtime.CucumberException: Expected step: "I login to testing as "testingUser"" got step: "I switch to the "System" tab"
    at cucumber.runtime.junit.JUnitReporter.fetchAndCheckRunnerStep(JUnitReporter.java:71)
    at cucumber.runtime.junit.JUnitReporter.match(JUnitReporter.java:61)
    at cucumber.runtime.Runtime.runStep(Runtime.java:278)
    at cucumber.runtime.model.StepContainer.runStep(StepContainer.java:44)
    at cucumber.runtime.model.StepContainer.runSteps(StepContainer.java:39)
    at cucumber.runtime.model.CucumberScenario.runBackground(CucumberScenario.java:59)
    at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:42)
    at cucumber.runtime.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:91)
    at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:63)
    at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:18)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:70)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:93)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:37)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:387)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
    at java.util.ArrayList$Itr.next(ArrayList.java:851)
    at cucumber.runtime.RuntimeOptions$1.invoke(RuntimeOptions.java:287)
    at com.sun.proxy.$Proxy17.feature(Unknown Source)
    at cucumber.runtime.junit.JUnitReporter.feature(JUnitReporter.java:165)
    at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:69)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:93)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:37)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:387)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
    at java.util.ArrayList$Itr.next(ArrayList.java:851)
    at cucumber.runtime.RuntimeOptions$1.invoke(RuntimeOptions.java:287)
    at com.sun.proxy.$Proxy17.uri(Unknown Source)
    at cucumber.runtime.junit.JUnitReporter.uri(JUnitReporter.java:160)
    at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:68)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:93)
    at cucumber.api.junit.Cucumber.runChild(Cucumber.java:37)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:387)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I'm not sure where to go from here.

Reporters in cucumber-jvm are not thread-safe, so this is not easily possible. You could try grouping tests using multiple runners and configuring parallel execution on class level. I you are feeling really adventurous you could try this reimplementation of cucumber: github.com/jhorstmann/zuchini (disclaimer: created by me) Documentation is still lacking at the moment but it should be mostly compatible and there are some examples in the repository. — Jörn Horstmann
another easier (not better) solution could be to use multiple runners which write to diifferent reports. each feature file can have an own runner... — Dude
have a look at stackoverflow.com/a/41100104/2895913 explained in detail. — Sugat Mankar

matt barbour matt barbour · Accepted Answer · 2015-11-25T15:30:33

Reporting is a real problem when trying to parallelize. We created a framework called zucchini which does a lot of the heavy lifting on parallel test execution and reporting for you and is totally compatible with cucumber-jvm. You might check it out and see if it works for your application

We addressed the reporting problem by using the json reporter and allowing each parallel thread to create their own independent json objects. We then munged all the json results together into one 'uber' json. You can see that code here

Parallel Cucumber-JVM Selenium Grid Tests using Maven Surefire

2 Answers