How to increase JMeter to handle 50,000 threads for distributed testing on Linux VMs?

0
votes

System:

I have the following JMeter setup for distributed, non-GUI testing: 1 Master VM, 2 Slaves. All 3 are Azure VMs: Ubuntu 16.04, 4 vcpus, 16GB RAM

I have followed this JMeter tuning guide and am not seeing improved results: https://www.blazemeter.com/blog/9-easy-solutions-jmeter-load-test-%E2%80%9Cout-memory%E2%80%9D-failure

I have also tuned all 3 systems according to the EMQ tuning guide: http://emqtt.io/docs/v2/tune.html

Problem: I am still unable to achieve 50,000 connections over 5000s (1h23m) with 2 Slave VMs.

According to the JMeter documentation, I can run 1000 threads in x amount of time per VM:

25.5 Limitations, 3. A single JMeter client running on a 2-3 GHz CPU (recent CPU) can handle 1000-2000 threads depending on the type of test.

https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.html

I am able to verify this in practice. I am also able to verify that I can achieve a maximum of 12,000 threads over 600s. (12,500 threads over any time results in JVM OOM Exception)

Since the target servers are VMs running EMQ instances, I will note that the official EMQ load test docs describe the test agent as being 1 VM 30000 VU (2 dockers*15000 VU. (http://emq-xmeter-benchmark-en.readthedocs.io/en/latest/connection.html#ssl-authentication)

Question: I am wondering if there is a known implementation of JMeter (+ Virtual Machines + VirtualBox or Docker or some other service) that allows for more threads. Or, if there is a known optimization for JMeter that I have not addressed?

1
multithreadingdockerjmeterjvmvirtualbox

1 Answers

1
votes

The number of virtual users JMeter can kick off depends on the nature of your test, to wit:

So don't believe the statement that a single JMeter instance can mimic 1000 threads. Neither believe the statement that it can handle 15000 threads. This is simply because they are not running your test under your hardware and software conditions. You can get > 100 000 users if your test plan will have a single Dummy Sampler with zero Connect, Latency and Response Times but this test is not representative.

  1. Set up monitoring of your baseline OS health metrics on the machines where JMeter Slaves are running, you can use i.e. JMeter PerfMon Plugin for this.
  2. Start your test and gradually increase the load at the same time keep an eye on the CPU, RAM, Network and Disk usage in the PerfMon
  3. When any of monitored metrics exceeds i.e. 90% of total available capacity stop your test and measure how many virtual users were active at this time - this is how many you will be able to simulate on particular this machine for particular this test type.
  4. If the number multiplied by the number of slave machines equals or higher than the total number of users you need to simulate - you're good to go. If not - you will need another slave machine.

With regards to your setup I can think only of running 3rd jmeter slave engine on the same machine where you have the master as JMeter master engine basically does nothing apart from sending .jmx script to slaves and collecting results. And if your master has the same hardware specs as slaves - you can utilise the remaining capacity to produce the higher load.