ApacheJMeter Throughput Shaping Timer very bad performance

Question

I'm using the Throughput Shaping Timer Plugin for ApacheJMeter to make a dynamic stress test for an adaptive application (application changes its behavior when resource demand changes).

My configuration (in attachment) is as follows:

seconds  0 -  4  : ramp up to 1000 requests per second (RPS)
seconds  5 -  14 : 1000 RPS
seconds  15 - 19 : ramp up to 5000 RPS
seconds  20 - 29 : 5000 RPS
seconds  30 - 34 : ramp up to 15000 RPS
seconds  35 - 44 : 15000 RPS
seconds  45 - 49 : ramp down to 5000 RPS
seconds  50 - 59 : 5000 RPS
seconds  60 - 64 : ramp down to 1000 RPS
seconds  65 - 74 : 1000 RPS
seconds  75 - 79 : ramp down to 0 RPS

For the execution of the test I use the following command on a Ubuntu 18 LTS with 16 Cores (each 3GHz) and 32GB RAM:

$ jmeter/bin/jmeter -n -t ~/JMeterTests/$1.jmx -l ~/TestResults/$2.jtl

Unfortunately, the CPU Usage skyrockets to 100% immediately when I start the test and many errors are produced by the requests:

Something is definitely wrong here since I used JMeter with this machine sending 10000 RPS without even reaching 10% CPU Usage. I can't figure out what is wrong - can anyone help?

Test File is in the attachment 3.

Thank you very much.

Answer 1

You have defined only 300 threads (virtual users) in the Ultimate Thread Group.

It means that you will be able to reach 15000 requests per second with 300 users only if the response time is 20 milliseconds or less.

From your summariser I can see average response time like 55 ms and maximum response times like 876 ms which means that these 300 users are definitely not enough even assuming the system under test is capable of handling that number of requests per second.

So recommendations are:

1. Make sure to follow JMeter Best Practices
2. Migrate to Concurrency Thread Group which can kick off extra threads if current amount is not enough to reach/maintain the desired throughput, it can be connected with the Throughput Shaping Timer via Feedback Function, check out Using JMeter’s Throughput Shaping Timer Plugin article for more details if needed.
3. The fact you used JMeter from this machine and had 10k requests per second for different test doesn't necessarily mean you will be able to produce 10k requests per second for the current test and future tests, it's a matter of JMeter configuration, OS configuration, test plan details, etc.

Answer 2

As this Article and this Question point out, the issue is resolved by changing the following parameters in the linux configuration:

a) Increased the local port range using below command:

echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range

b) Enable reuse of sockets as below:

sudo sysctl -w net.ipv4.tcp_tw_reuse=1

Note that the hints Dmitri T. gave above should be considered. Using Concurrency Thread Group combined with the Feedback Function benefits the overall test plan. Next, the changes made to the configuration may not be healthy for a production system. I cannot make any educated statement about that. Finally, note that you have to apply the configuration changes in all test servers in a distributed testing environment.

EDIT:

Note as well, that the maximum value of users is still limited after the configuration, it is just higher than before.