What constitutes a true error in JMeter when the server always returns 200?

1
votes

I am load testing an e-commerce site. Under medium to heavy loads, Jmeter reports an unusually high number of errors, i.e. succes=FALSE in the resulting .csv file which translates to a fairly high % errors in the JMeter dashboard report.

On inspecting the Kibana logs, other than a few WARN, there aren't any errors and the status for all the http requests are 200. In fact, I am able to verify that the test users created in my flow are able to login and view whatever the test was supposed to do.

My question is - how is JMeter determining this error (non-200 ?) when the server side returns nothing but 200? For fewer threads, the errors are 0%.

1
jmeterload-testing
Were there any timeouts? – user7294900
There were no timeouts – Jai
It can be marked as failed due to assertions, mainly duration assertions – user7294900
I dont even have duration assertions in my script. It is definitely a function of the higher load but I am unable to correlate since the logs have no errors – Jai

1 Answers

1
votes

First I would check if any assertions can fail for any reasons. So disabling any assertions and running most primitive version of the script without any possibility of failure (other than based on sampler response) would be my first step. If it doesn't help...

HTTP Sampler will fail for return code < 200, or return code >= 400, or any Java exception, which will happen when request could not be sent, or response from the server was not received (in full). Most commonly those are timeouts related to various underlying problems:

  • Lack of available ports on JMeter side. Since you indicated that problem happens only with more JMeter users, it is a possibility. On paper you have 65535 ports, but both Windows and Linux are limiting number of available ports to few thousands by default. Make sure you have enough ports to run the load, and monitor port usage as you ramp threads up.

  • Even if all ports are available, many ports may be sitting for a few minutes in TIME_WAIT or CLOSE_WAIT state, depending on how your client and server interact and other network issues. So theoretically you may have 65k ports, but practically you don't have enough. In this case it's worth checking why it happens (can point to a bug) and possibly reduce the time ports will be waiting in this *_WAIT state.

  • You also need to make sure you have enough RAM, JVM heap and CPU: monitor to see if there's a bottleneck that can cause such slowness that breaks the ability of a Sampler to finish its work successfully.

If all of the above is working as expected, then either server or someone between client and server (load balancer, proxy, firewall) is causing this issue.

  • I would start from the server, and verify that all requests you are sending are received by a server. E.g. if you send 2000 requests, and server receives 1000, which finish successfully, then from server perspective everything is fine, but Jmeter will be at 50% error rate.

  • Also check your JMeter log definitions and make sure failures and exceptions are written to them properly (they are multi-line messages, and may cause log definitions to break when getting to them, and thus you wont' see them). For example run HTTP Sampler that points to non-available server. See if you can diagnose what returned exception was.

Beyond that - figuring out which part is "lying" to you would be an approach, but hard to offer any more specific advice based on current information