We are running a node API application on Elastic Beanstalk, and are using Dynamo for a DB, and Redis in Elastic Cache. I'm loading testing, by running a loop of request calls against the node API, simulating about 20,000 simultaneous users making calls.
It runs fine for a while. It gets up to just over 30,000 total calls. Then I start not being able to connect to dynamo and redis. I get the following:
"Inaccessible host: dynamodb.us-west-2.amazonaws.com'. This service may not be available in theus-west-2' region.\",\"code\":\"UnknownEndpoint\",\"stack\":\"UnknownEndpoint: Inaccessible host: `dynamodb.us-west-2.amazonaws.com'."
"Redis connection to xxxxxx-lt-001.19atpm.0001.usw2.cache.amazonaws.com:6379 failed - getaddrinfo ENOTFOUND"
But it's already been running quite happily for quite a while at that point.
I see around 20,000/30,000 connections on the app server. I see a response time of less than 200 ms for my calls up to that point, and all of a sudden they jump in length, getting steadily longer and longer. I have upped the open file count on the app server to 100,000, and the max connections for node to 50,000.
The app server is a C4.xlarge. CPU is not hitting 30%. Neither Dynamo nor Redis are near their limits, far from it. Network out peaks at 40,000,000, Network in peaks at 100,000,000.
I'm now completely stumped. Any suggestions?
/etc/security/limits.conf file:
- soft nofile 100000
- hard nofile 100000
Results of ulimit -a
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 59742 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 100000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 59742 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Bit more info - I found some errors in the messages log about "nf_conntrack: table full, dropping packet." So I followed the suggestions in the following web page to increase the size of the table. That got me further, up to 45,000 or so calls. Then it really fell off the table. Went immediately from subsecond response time to 25 second response time.
