AWS EC2 Instance Becomes Unresponsive

Question

I am new to AWS platform and started using it 3 months back. I launched a EC2 instance which was a c4.xlarge instance type of Linux. I am running an application called SPLUNK on that instance for logging and monitoring. From the past 1 month I am observing splunk web interface stops responding every now and then and when I try to SSH into instance, that also becomes unresponsive. After that it takes from 5min - 2 hours to come back online until I go ahead and manually start and stop the instance. I searched through the available forums and thought may be my instance is under a lot of load since I was seeing spikes of 100% so I upgraded the instance to C4.2xlarge. But unfortunately issue still persists even when the Average CPU load is around 35% now and sometimes when multiple people are using splunk then Max spike goes to 80%. EC2 status checks still shows everything is fine but instance itself becomes unresponsive. Any help would be much appreciated on this issue.

Thanks Saurabh

Answer 1

This could be a memory issue.

• Are you able to capture memory usage of the various services on the ec2 and spool them to a separate logging service?
• Have you inspected the logs for the splunk service to check for errors that indicate inability to allocate memory or access system resources of any kind?

Ironically, you need to add some monitoring of your monitoring application ;)