JMeter doesn't fail on BeanShell errors

Question

In my company we use JMeter as part of our release pipeline to run various tests (smoke, load, etc). As now I'm creating more complicated test scenarios, I'm using a lot of BeanShell and I'm concerned by the fact that JMeter seems to be totally fine with failed BeanShell scripts, and I'd like to find a way to fail tests when BeanShell fails.

If a syntax error is present, the script will fail, report to the log file, but it won't prevent the whole test suite to succeed, as long as the missing BeanShell execution don't trigger anything else to fail. Example (click for full size image):

As seen above, I have a sampler with a BeanShell postprocessor that contains a syntax error. When executed the error logged and increases the error counter on the top, but that's all. If the user doesn't notice that, the error will slip to our release process, making our tests less reliable. If I run JMeter with the command line, this is the result:

Writing log file to: /home/user/jmeter/jmeter.log
Creating summariser <summary>
Created the tree successfully using Bad BeanShell Test.jmx
Starting the test @ Fri Feb 24 12:23:11 CET 2017 (1487935391815)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary =      2 in 00:00:01 =    1.9/s Avg:   382 Min:    29 Max:   736 Err:     0 (0.00%)
Tidying up ...    @ Fri Feb 24 12:23:13 CET 2017 (1487935393168)
... end of run

Again, seems all right, only the log file to find out something is wrong. And adding a post-test script to look for errors on the file is not a good self-contained test case.

I tried a BeanShell Listener but it doesn't seem to expose the results of the script compilation/execution (SampleEvent doesn't contain useful info), and even after a lot of searching, I found no way to do it at all.

It could be a listener, an assertion, custom Java code on a plugin, a command line configuration, anything as long as I can add it once on my test suite, and at least trigger one error or abort the suite to ensure it will be investigated.

Dmitri T Dmitri T · Accepted Answer · 2017-02-24T12:43:52

The only way to get Beanshell PostProcessor failure in the test results is amending parent sampler result setting it to "Failure" at the Beanshell script beginning and to "Success" at the script failure, something like:

prev.setSuccessful(false);
//your main Beanshell code goes here
prev.setSuccessful(true);

So if any Beanshell error will occur in-between these prev.setSuccessful methods - the parent sampler will be failed.

Given your test relies on scripting I would recommend switching to JRS223 Test Elements and Groovy language, it will be much better from performance perspective, see Groovy Is the New Black article for more information, benchmarks, and scripting best practices.

JMeter doesn't fail on BeanShell errors

1 Answers