Create CloudWatch alarm that sets an instance to standby via SNS/Lambda

Question

What I am looking to do is set an instance to standby mode when it hits an alarm state. I already have an alarm set up to detect when my instance hits 90% CPU for a while. The alarm currently sends a Slack and text message via SNS calling a Lambda function. I would like to add is to have the instance go into standby mode. The instances are in an autoscaling group.

I found that you can perform this through the CLI using the command :

aws autoscaling enter-standby --instance-ids i-66b4f7d5be234234234 --auto-scaling-group-name my-asg --should-decrement-desired-capacity

You can also do this with boto3 :

    response = client.enter_standby(
        InstanceIds=[
            'string',
        ],
        AutoScalingGroupName='string',
        ShouldDecrementDesiredCapacity=True|False
    )

I assume I need to write another Lambda function that will be triggered by SNS that will use the boto3 code to do this?

Is there a better/easier way before I start?

I already have the InstanceId passed into the event to the Lambda so I will have to add the ASG name in the event.

Is there a way to get the ASG name in the Lambda function when I already have the Instance ID? Then I do not have to pass it in with the event.

Thanks!

Answer 1

Your question has a couple sub-parts, so I'll try to answer them in order:

I assume I need to write another Lambda function that will be triggered by SNS that will use the boto3 code to do this?

You don't need to, you could overload your existing function. I could see a valid argument for either separate functions (separation of concerns) or one function (since "reacting to CPU hitting 90%" is basically "one thing").

Is there a better/easier way before I start?

I don't know of any other way you could do it, other than Cloudwatch -> SNS -> Lambda.

Is there a way to get the ASG name in the Lambda function when I already have the Instance ID?

Yes, see this question for an example. It's up to you whether it looks like doing it in the Lambda or passing an additional parameter is the cleaner option.

Answer 2

For anyone interested, here is what I came up with for the Lambda function (in Python) :

# Puts the instance in the standby mode which takes it off the load balancer
#   and a replacement unit is spun up to take its place
#

import json
import boto3

ec2_client = boto3.client('ec2')
asg_client = boto3.client('autoscaling')

def lambda_handler(event, context):
    # Get the id from the event JSON
    msg = event['Records'][0]['Sns']['Message']
    msg_json = json.loads(msg)
    id = msg_json['Trigger']['Dimensions'][0]['value']
    print("Instance id is " + str(id))

    # Capture all the info about the instance so we can extract the ASG name later
    response = ec2_client.describe_instances(
        Filters=[
            {
                'Name': 'instance-id',
                'Values': [str(id)]
            },
        ],
    )

    # Get the ASG name from the response JSON
    #autoscaling_name = response['Reservations'][0]['Instances'][0]['Tags'][1]['Value']
    tags = response['Reservations'][0]['Instances'][0]['Tags']
    autoscaling_name = next(t["Value"] for t in tags if t["Key"] == "aws:autoscaling:groupName")
    print("Autoscaling name is - " + str(autoscaling_name))

    # Put the instance in standby
    response = asg_client.enter_standby(
        InstanceIds=[
            str(id),
        ],
        AutoScalingGroupName=str(autoscaling_name),
        ShouldDecrementDesiredCapacity=False
    )