Jenkins validates the JMeter build as successful when it’s actually failed

Question

I have similar issue, that blocks me. I need to run my JMeter tests with Jenkins. But Jenkins validates the JMeter build as successful when it’s actually failed. I wonder what I am doing wrong so that the jmeter won't return fail when an assertion fails. I run a Windows Batch script calling jMeter. Here's how:

COMMAND LINE

cd C:\apache-jmeter-3.1\bin
jmeter -n -c -t C:\Users\maria\Desktop\Automation\WIP\Test-page.jmx -l C:\Users\maria\Desktop\Automation\WIP\Test-page.xml

And saves the results in .xml to validate the report with the Performance plugin (v 2.0) and Jenkins ver. 2.32.1

CONSOLE

Created the tree successfully using C:\Users\maria\Desktop\Automation\WIP\Test-page.jmx
Starting the test @ Fri Dec 30 11:05:58 GMT 2016 (1483095958952)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary =      1 in 00:00:00 =    3.1/s Avg:   206 Min:   206 Max:   206 Err:     1 (100.00%)
Tidying up ...    @ Fri Dec 30 11:05:59 GMT 2016 (1483095959326)
... end of run


Performance: Recording JMeter reports 'C:\Users\maria\Desktop\Automation\WIP\Test-page.xml'
Performance: Parsing JMeter report file 'C:\Jenkins\jobs\Test demo\builds\3\performance-reports\JMeter\Test-page.xml'.
Performance: Percentage of errors greater or equal than 0% sets the build as unstable
Performance: Percentage of errors greater or equal than 0% sets the build as failure


Finished: SUCCESS

As you can see I have 1 error. But the Performance plugin sets the build to success.

PLUGIN SETTINGS performance plugin settings

Jmeter properties

jmeter.save.saveservice.assertion_results=all
jmeter.save.saveservice.output_format=xml
jmeter.save.saveservice.response_data=true

TEST CASE Test-page.jmx

Attempts I tried and didn’t work for me:

1. older version of the Performance plugin (v 1.11 and v.16)
2. .jtl format
3. .xml
4. .csv
5. adding "echo" to the command line in the end
6. setting jmeter.save.saveservice.response_data=true

Ran out of ideas how to make the Performance plugin count the errors and raise the "Failed" flag :(

Please help.

Answer 1

Set "Use error thresholds on single build" from 0 to 1.

This should mean that the jenkins build fails if at least 1 error is in your JMeter test case, where a 0 ignores the error count (If I remember this correctly).

Answer 2

Sounds like a Performance Plugin issue, I believe you should report it via Jenkins Bug Tracker

In the meantime to work this around I would suggest returning non-zero exit code for the JMeter build step, the fastest and the easiest way is running your JMeter test using Taurus tool as a wrapper. Taurus has powerful Pass/Fail Criteria subsystem where you can define the conditions of your build failure. If the condition will be met Taurus execution step will be finished with code 3. Jenkins is smart enough to automatically fail command-line tasks with non-zero exit code.

Minimal working Taurus config file:

---
execution:
  scenario: my-test

scenarios:
  my-test:
    script: C:\Users\maria\Desktop\Automation\WIP\Test-page.jmx

services:
- module: passfail
  criteria:
  - succ<100%, stop as failed

More information, if needed: Meetup Recap on Using Taurus to Automate JMeter and Jenkins Tests

You can also change JMeter exit code directly form the JMeter test via i.e. JSR223 Scripting Elements like:

if (some condition) {
    System.exit(1);
}

but in this case you won't have much flexibility in terms of setting failure criteria

Answer 3

I set my Select evaluation mode to standard mode, Set the radio button on Error Threshold and set Unstable and Failed to 1