Locust - Hatch Rate - EC2

Question

I am using locust to load test an application. I wrote and tested the script on my local ubuntu system, and all went well.

I created an EC2 instance, using an Amazon Linux image, and after adjusting the file limits in /etc/security/limits.conf file I loaded up locust and things went normally for a small test (simple GET test, just to check the plumbing, 2000 users, 20 hatch rate).

However, when I loaded up a larger test, 8000 users 40 hatch rate, I noticed that somewhere around 3,000 or 4,000 users, the hatch rate appeared to slow down, just adding 4 - 5 rather the 40 new "users" at a time. So it took a long time to reach 8000. Is that expected behavior, if not, any idea what the problem might be?

Answer 1

What Locust calls "users" is actually gevent spawned TaskSets. This means you're spawning thousands of eventlets in a single Python process, which means a great deal of overhead managing those eventlets.

If you want to spawn thousands of TaskSets, I'd recommend running Locust in distributed mode. You can have many slaves running on the same hardware, or distribute your slaves across many instances. Google has written up a neat article and open sourced some Kubernetes containers for just such a purpose. We wrote our own Docker container with Alpine and a heavily modified Locust, our ratio of slaves to TaskSets ended up being 1:100. The ratio of slaves to instances heavily depends on what instance size you get.