JMeter and Jenkins: Enabling Jenkins Status to Turn Red When Assertion Is Not Met

Question

I'm still somewhat new to JMeter, but here's my issue:

After checking in my JMeter API test plan into Jenkins, when Jenkins runs the test (via a .bat file) and an Assertion fails (expected = 200, but actual != 200), the test knows to drop out of the thread group and go on to the next thread. When the test is done running, Jenkins shows a status of Green for success even though in theory the test fails. (maybe I'm handling the error too gracefully?) The following is shown on the Jenkins Console window that shows a Green status for the test.

Creating summariser <summary>
Created the tree successfully using API Test Plan.jmx
Starting the test @ Thu Feb 18 09:09:41 EST 2016 (1455804581076)
Waiting for possible shutdown message on port 4445
summary +      1 in     1s =    1.0/s Avg:     7 Min:     7 Max:     7 Err:     1 (100.00%) Active: 1 Started: 1 Finished: 0
summary +      2 in   0.1s =   16.3/s Avg:     1 Min:     1 Max:     2 Err:     2 (100.00%) Active: 0 Started: 1 Finished: 1
summary =      3 in   1.1s =    2.7/s Avg:     3 Min:     1 Max:     7 Err:     3 (100.00%)
Tidying up ...    @ Thu Feb 18 09:09:42 EST 2016 (1455804582362)
... end of run

I'm being asked to make the Jenkins status turn red when this happens. One way I thought of was to have the .bat file that invokes the JMeter test plan was to have an Assertion write an error file, and then in the .bat file state if the file exists, set errorlevel = 1 and exit. However, I'm not being successful in generating the error file.

Any insight how to fix my file generation and thoughts on how to approach the Jenkins status issue is very (very) much appreciated.

Here's and outline of a Thread Group

Thread Group 1
    Properties:
        Number of Thread (users) = 1
        Ramp-Up Period (in seconds) = 5
        Loop Count = 10
        Delay Thread creation until needed = enabled
        Scheduler = disabled

        Loop Controller 1
        Loop Counter 1              
            HTTP Request: CREATE new database record
                RegEx: (RefName = newRequest)
                Response Assertion
                    Main:Response Code:Equals:200
                BeanShell Assertion
                    Param: ${__setProperty(newRequest,${newRequest},)}
                    Script:
                        if (ResponseCode.equals("200") == true) {
                            SampleResult.setResponseOK();  
                        }
                        else if (ResponseCode != null && ResponseCode.equals ("200") == false ) { 
                            Failure = true;
                            FailureMessage ="Response code was not a 200 response code it was " + ResponseCode + "." ; // displays in Results Tree
                            print ( "Creation of a new record failed: the return code is " + ResponseCode + ".");   // goes to stdout
                            log.warn( "the return code is " + ResponseCode); // this goes to the JMeter log file but i want it to go to an error.log file
                        }

Konstantin Dobroliubov Konstantin Dobroliubov · Accepted Answer · 2016-02-19T15:29:44

I can suggest you to use Jenkins Performance Plugin and run your tests via ant, not as .bat file. This plugin do know how to mark your job as FAILURE in different cases. For example, you can configure your job to be marked as red in case of any error.

JMeter and Jenkins: Enabling Jenkins Status to Turn Red When Assertion Is Not Met

1 Answers