AWS SNS, Lambda and AutoScaling timing issues

Question

I am using Terraform for AWS deployments and currently I am trying to tie a Lambda function to the scaling behavior of an ECS cluster. This works in general, but the timing of things is unacceptable. In my latest attempt, the cluster grows at 1:11pm, 1:14pm, 1:17pm and at 1:20pm, but the Lambda function is triggered at 1:11pm, 1:36pm, 1:38pm and 1:56pm.

I am looking for a solution where the Lambda function is triggered (about) when the cluster scales, i.e. spawns addional EC2 instances).

My approach works like this:

The ECS cluster consists of a single aws_autoscaling_group and a single ELB.
For the autoscaling group I created an aws_autoscaling_policy with
- adjustment_type = "ChangeInCapacity" and
- scaling_adjustment = "5"
There is a dedicated aws_sns_topic for the scaling of this cluster
I trigger the autoscaling group and publish to the SNS topic within the same aws_cloudwatch_metric_alarm:

:

resource "aws_cloudwatch_metric_alarm" "ecs_grow" {
    [...]
    comparison_operator = "GreaterThanOrEqualToThreshold"
    namespace = "AWS/ECS"
    metric_name = "CPUUtilization"
    threshold = "16"
    statistic = "Average"
    period = "60"
    evaluation_periods  = "1"
    alarm_actions = [
        "${aws_autoscaling_policy.grow_policy.arn}",
        "${aws_sns_topic.scaling_topic.arn}"
    ]
}

The Lambda function currently only writes the event into a log.

With this setup I generate load on my cluster so that it scales out every 3 minutes. I can validate that this works by looking at the Cloudwatch metrics GroupDesiredCapacity GroupTotalInstances and of course the EC2 instances the AWS console shows me. Indeed, the cluster grows every 3 minutes by 5 instances.

I started out with 5 instances and let the cluster scale 4 times. This means at the end I had a cluster with a total of 25 instances. In my Cloudwatch metrics I can see the GroupDesiredCapacity graph climb by 5 at 1:11pm, 1:14pm, 1:17pm and at 1:20pm, just as expected and in accordance with what I can see on the AWS console.

My problem is, that the Lambda function is triggered only eventually. I get log entries at 1:11pm, 1:36pm, 1:38pm and 1:56pm.

What really confuses me is that the StateChangeTime reported by the alarms are 1:11pm, 1:36pm, 1:38pm and 1:56pm. So it would appear that the Lambda function is indeed triggered as soon as the messages are published.

Where does this mismatch between the triggering of the autoscale policy and the message publication come from? More importantly, how do I align the two?

I agree that it's strange to get the lambda triggered so late when the cluster scales so quickly. Question though... why don't you scale up on the cluster CPU? That's probably what you actually care about keeping low. — Eric M. Johnson
@EricJohnson I am sorry, I do not understand what you mean by your question. My alarm triggers its actions w.r.t monitoring data from AWS/ECS CPUUtilization. I thought this already is "scaling on the cluster CPU". — user1129682
You didn't specify which resource you were scaling on, so I assumed you were scaling on the service. I was not aware there is a cluster level CPU metric on which you can create alerts. Using the cluster CPU (as opposed to my suggestion which was utilizing the CPU of the AutoScale Group) seems valid. — Eric M. Johnson

user1129682 user1129682 · Accepted Answer · 2018-07-06T09:27:07

I found a workaround. I am using an autoscaling notification to send other/more notifications:

resource "aws_autoscaling_notification" "scaling_notifications" {
  group_names = [ "${aws_autoscaling_group.ecs.name}" ]
  notifications = [ "autoscaling:EC2_INSTANCE_LAUNCH" ]
  topic_arn = "${aws_sns_topic.scaling_topic.arn}"
}

For some reason this triggers the lambda function when and as often as I want it to. Notice, it is the same SNS topic and the same lambda function as before!

AWS SNS, Lambda and AutoScaling timing issues

1 Answers