AWS Cloudwatch Heartbeat Alarm

13
votes

I have an app that puts a custom Cloudwatch metric to AWS every minute. This is supposed to act as a heartbeat so I know the app is alive.

Now I want to put an alarm on this metric to notify me if the heartbeat stops. I have tried to accomplish this using different cloudwatch alarm statistics including "average" and "data samples" and setting an alarm threshold less than 1 over a given period. However, in all cases, if my app dies and stops reporting the heartbeat, the alarm will only go into an "Insufficient Data" state and never into an "Alarm" state.

I understand I can put a notification on the "Insufficient Data" state, but I want this to show up as an alarm. Is this possible in Cloudwatch?

Thanks,

Matt

3
amazon-web-servicesamazon-cloudwatch
Insufficient data should be considered an actionable alarm. Why is that a problem for you? - tedder42
Causes theres a difference when someone sees insufficient data vs. alarm - Levitron
There shouldn't be. If servers/monitors/data goes missing that should be a substantial issue. - tedder42
Sure. But then the alarm exists for the sole purpose of going to the insufficient data state. It will never go to the alarm state. You can use any metric and any threshold (it will meet) and you don't care. You just care if it stops receiving metrics. It just seems like a hack rather than building an alarm for your exact use case. - Levitron
Did you ever solve this problem? I have it too. - xpmatteo

3 Answers

5
votes

I think that the alarm going into "Insufficient Data" state has to do with how missing data is being handled. As the doc states:

Similar to how each alarm is always in one of three states, each specific data point reported to CloudWatch falls under one of three categories:

  • Not breaching (within the threshold)
  • Breaching (violating the threshold)
  • Missing

You can specify how alarms handle missing data points. Choose whether to treat missing data points as:

  • missing (The alarm looks back farther in time to find additional data points)
  • notBreaching (Treated as a data point that is within the threshold)
  • breaching (Treated as a data point that is breaching the threshold)
  • ignore (The current alarm state is maintained)

The default behavior is missing.

So i guess that specifying missing data points as breaching would do the trick :)

4
votes

Instead of pushing in a custom metric to Cloudwatch, consider:

Push a message onto an SNS topic, on the same periodic basis as you were doing, and set up a CloudWatch monitor for the SNS topic's NumberOfMessagesPublished metric. If the number of heartbeats falls below the expected value for the time period you specify, whether its because the app crashed, or server crashed, the metric will go into an Alarm state.

4
votes

Treat missing data as breaching threshold (step 4)

Check this: https://cloudonaut.io/dead-mans-switch-with-cloudwatch/

enter image description here