How to propagate the Jmeter regression failures to Jenkins Build

Question

I have a maven project that runs some regression test cases (NOT load tests) through JMeter.

Here I have setup this project to be executed in Jenkins.

Maven. I am using the jmeter-maven-plugin(v2.0.3), to execute the Jmeter tests.

Jmeter Jmeter 3.0 (ApacheJMeter-3.0.jar though jmeter-maven-plugin) The project has been set up following the same method stated in this page.

Jenkins. The project is configures as a Free style project on Jenkins(v1.618).

When I execute the project, the test gest executed and the dashboard also gets generated.The build completes with [INFO] BUILD SUCCESS.

Since the Maven Build is successful , Jenkins always shows the build as a successful build. Even when there are JMtere tests failing, Jenkis mark the build as success.

How can I propagate the JMtere regression test failures to Jenkins?

I don't need to use and load test reporting plugin as these are not meant to be load tests and just regression tests.

The relevant parts of my build log is shown below.

[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building TestProj Regression Test Suite 0.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ TestProj-regression ---
[INFO] Deleting C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target (includes = [**/*.csv, **/*], excludes = [])
[INFO] 
[INFO] --- maven-clean-plugin:3.0.0:clean (Deleting all files under target, but not target itself) @ TestProj-regression ---
[INFO] Deleting C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target (includes = [**/*.csv, **/*], excludes = [])
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-resources) @ TestProj-regression ---
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 11 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ TestProj-regression ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\Testuser\Documents\TestProj\TestProj_Regression\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ TestProj-regression ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ TestProj-regression ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 697 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ TestProj-regression ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ TestProj-regression ---
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ TestProj-regression ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target\TestProj-regression-0.1.0-SNAPSHOT.jar
[INFO] 
[INFO] >>> jmeter-maven-plugin:2.0.3:jmeter (execute-jmeter-tests) > :configure @ TestProj-regression >>>
[INFO] 
[INFO] --- jmeter-maven-plugin:2.0.3:configure (configure) @ TestProj-regression ---
[INFO] -------------------------------------------------------
[INFO]  Configuring JMeter...
[INFO] -------------------------------------------------------
[INFO] 
[INFO] <<< jmeter-maven-plugin:2.0.3:jmeter (execute-jmeter-tests) < :configure @ TestProj-regression <<<
[INFO] 
[INFO] --- jmeter-maven-plugin:2.0.3:jmeter (execute-jmeter-tests) @ TestProj-regression ---
[INFO]  
[INFO] -------------------------------------------------------
[INFO]  P E R F O R M A N C E    T E S T S
[INFO] -------------------------------------------------------
[INFO] Invalid value detected for <postTestPauseInSeconds>.  Setting pause to 0...
[INFO]  
[INFO]  
[INFO] Executing test: TestProj_Regression.jmx
[INFO] Writing log file to: C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target\jmeter\logs\TestProj_Regression.jmx.log
[INFO] Creating summariser <summary>
[INFO] Created the tree successfully using C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target\jmeter\testFiles\TestProj_Regression.jmx
[INFO] Starting the test @ Tue Dec 13 17:03:33 AEDT 2016 (1481609013485)
[INFO] Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
[INFO] Base Dir root [C:/Users/Testuser/Documents/TestProj/TestProj_Regression/target/jmeter]
[INFO] DEbug line 3
[INFO] envFlag:           [TestEnv1]
[INFO] baseDir:           [C:/Users/Testuser/Documents/TestProj/TestProj_Regression/target/jmeter]
[INFO] dataFilesPath:     [C:/Users/Testuser/Documents/TestProj/TestProj_Regression/target/jmeter/TestProjRegression/DataFiles/TestEnv1]
[INFO] runConfigFilesPath:[C:/Users/Testuser/Documents/TestProj/TestProj_Regression/target/jmeter/TestProjRegression/RunConfig]
[INFO] outputFilesPath:   [C:/Users/Testuser/Documents/TestProj/TestProj_Regression/target/jmeter/TestProjRegression/OutputFiles/TestEnv1]
[INFO] resultsPath:       [C:/Users/Testuser/Documents/TestProj/TestProj_Regression/target/jmeter/TestProjRegression/Results/TestEnv1]
[INFO] Config File [C:/Users/Testuser/Documents/TestProj/TestProj_Regression/target/jmeter/TestProjRegression/RunConfig/RunConfig_TestEnv1.csv]
[INFO].
[INFO].
[INFO].
[INFO].
[INFO].
[INFO] summary +      1 in 00:00:00 =    4.2/s Avg:   149 Min:   149 Max:   149 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
[INFO].
[INFO].
[INFO].
[INFO].
[INFO] summary +      1 in 00:00:18 =    0.1/s Avg:  7496 Min:  7496 Max:  7496 Err:     1 (100.00%) Active: 0 Started: 1 Finished: 1
[INFO] summary =      2 in 00:00:18 =    0.1/s Avg:  3822 Min:   149 Max:  7496 Err:     1 (50.00%)
[INFO] Tidying up ...    @ Tue Dec 13 17:03:52 AEDT 2016 (1481609032067)
[INFO] ... end of run
[INFO] Completed Test: TestProj_Regression.jmx
[INFO] 
[INFO] --- maven-antrun-plugin:1.3:run (default) @ TestProj-regression ---
[INFO] Executing tasks
    [mkdir] Created dir: C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target\jmeter\results\dashboard
     [copy] Copying 1 file to C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target\jmeter\bin
     [copy] Copying 696 files to C:\Users\Testuser\Documents\TestProj\TestProj_Regression\target\jmeter\bin\report-template
     [java] Writing log file to: C:\Users\Testuser\Documents\TestProj\TestProj_Regression\jmeter.log
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:57 min
[INFO] Finished at: 2016-12-13T17:04:13+11:00
[INFO] Final Memory: 18M/185M
[INFO] ------------------------------------------------------------------------
[INFO] Shutdown detected, destroying JMeter process...

Answer 1

The fastest and the easiest way is using Performance Plugin, it has capabilities to define error thresholds on single build or on historical results

Moreover you will have "Performance Trend" charts on your build dashboard so you will be able to track the (hopefully) positive dynamic of your application performance.

Alternative way is using Taurus tool as a wrapper for your JMeter test instead of Maven, it provides powerful and flexible Pass/Fail Criteria subsystem where you can specify failure conditions. If the specified threshold is (are) exceeded - Taurus will simply return non-zero exit code and Jenkins is smart enough to treat it as a step failure.