Remote windows machines(Amazon) disconnected from network when running 4500 users test using JMeter

0
votes

I am running 4500 users test using JMeter in 2 distributed machines. Following are the machine details.

Windows Slave Machine details: Windows server R2

RAM: 8GB

CPU: 4 core

HEAP size which is configured in Jmeter.bat file

set HEAP=-Xms1024m -Xmx10240m set NEW=-XX:NewSize=512m -XX:MaxNewSize=2048m

Now here the problem is once test reached to 4500 users remote machines(Amazon windows instances) are getting disconnected from the network. Unable to run the script from Master.

Please provide me the solution do I need to change any other configurations.

Thanks in Advance.

2
performancejmeterperformance-testing
Amazon Instance Status check is failing. – daitha shankar
Thank u all for the suggestions, Issue got fixed after updating drivers on windows server, Its a problem from the AWS machines. – daitha shankar

2 Answers

0
votes

Most likely you simply overload your instances, for example you're setting maximum heap to 10 GB on a machine having 8GB of total available physical RAM which is not something recommended as:

  • Windows Server consumes some RAM itself
  • If you set maximum heap to something above your free RAM it will trigger swapping which will horribly slow down your test

So I would recommend gradually increasing the number of virtual users and at the same time monitoring your JMeter slaves using i.e. Amazon CloudWatch or JMeter PerfMon Plugin and keep an eye on at least the following metrics:

  1. CPU
  2. RAM
  3. Network IO
  4. Disk IO

If any of them reaches threshold of, say, 90% - check the number of active JMeter Threads at this moment using i.e. Active Threads Over Time listener - this is how many virtual users you can mimic. It might be the case you will need to add more slave nodes in order to reach 4500 virtual users.

Remember, each JMeter slave is actually independent so if you define 4500 virtual users in Thread Group and you will have 2 slaves the total load will be 4500 x 2 == 9000 virtual users, for 3 remote slaves it will be 4500 x 3 == 13500, etc.

0
votes

I wonder what kind of instances are you using for that. I may overlook it, of course, but when I look at the table, I don't see any combination like 4 vCPU - 8Gb mem?

Though whatever it is - check the network performance of your instance(s). If it is lower than High, especially if it is Low to Moderate - first, you may saturate it quick, it's not a lot, actually, 350-400 Mbps (I don't know your payload/rps rate, though, so calculate yourself at which point you'd hit the limit). And second - it might have been throttled after some period at full saturation, then your network performance get dropped down almost by half. I know, Amazon didn't document it (they only document CPU throttling on burstable performance instances, like t2), although I've seen it myself very clearly.

So, in addition to what Dmitri suggest, check the network throughput at your machines, right as your tests are running, live. You may just see it right away.

And the second suggestion - increase the number of slaves, at least by two, and spread load among them: this way, you won't hit that limit.