Terraform: configuring cloudwatch log subscription delivery to lambda?

Question

I need to ship my cloudwatch logs to a log analysis service.

I've followed along with these articles here and here and got it working by hand, no worries.

Now I'm trying to automate all this with Terraform (roles/policies, security groups, cloudwatch log group, lambda, and triggering the lambda from the log group).

But I can't figure out how to use TF to configure AWS to trigger the lambda from the cloudwatch logs.

I can link the two TF resources together by hand by doing the following (in the Lambda web console UI):

go into the lambda function's "Triggers" section
click "Add Trigger"
select "cloudwatch logs" from the list of trigger types
select the log group I want to trigger the lambda
enter a filter name
leave the filter pattern empty (implying trigger on all log streams)
make sure "enable trigger" is selected
click the submit button

Once that's done, the lambda shows up on the cloudwatch logs console in the subscriptions column - displays as "Lambda (cloudwatch-sumologic-lambda)".

I tried to create the subscription with the following TF resource:

resource "aws_cloudwatch_log_subscription_filter" "cloudwatch-sumologic-lambda-subscription" {
  name            = "cloudwatch-sumologic-lambda-subscription"
  role_arn        = "${aws_iam_role.jordi-waf-cloudwatch-lambda-role.arn}"
  log_group_name  = "${aws_cloudwatch_log_group.jordi-waf-int-app-loggroup.name}"
  filter_pattern  = "logtype test"
  destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}

But it fails with:

aws_cloudwatch_log_subscription_filter.cloudwatch-sumologic-lambda-subscription: InvalidParameterException: destinationArn for vendor lambda cannot be used with roleArn

I found this answer about setting up a similar thing for a scheduled event, but that doesn't seem to be equivalent to what the console actions I described above do (the console UI method doesn't create an event/rule that I can see).

Can someone give me a pointer on what I'm doing wrong please?

It's hard to see exactly but it looks like AWS is saying that the role you have given the PutSubscriptionFilter doesn't have access to Lambda. Can you also post the definition of the aws_iam_role.jordi-waf-cloudwatch-lambda-role.arn resource? — ydaetskcoR
Just make sure your destination_arn of aws_cloudwatch_log_subscription_filter resource does not include anything like ":*" in end. I messed up and spent whole day solving the issue. — Rashmi Jain

Shorn Shorn · Accepted Answer · 2016-07-18T04:57:22

I had the aws_cloudwatch_log_subscription_filter resource defined incorrectly - you should not provide the role_arn argument in this situation.

You also need to add an aws_lambda_permission resource (with a depends_on relationship defined on the filter or TF may do it in the wrong order).

Note that the AWS lambda console UI adds the lambda permission for you invisibly, so beware that the aws_cloudwatch_log_subscription_filter will work without the permission resource if you happen to have done the same action before in the console UI.

The necessary TF config looks like this (the last two resources are the relevant ones for configuring the actual cloudwatch->lambda trigger):

// intended for application logs (access logs, modsec, etc.)
resource "aws_cloudwatch_log_group" "test-app-loggroup" {
  name              = "test-app"
  retention_in_days = 90
}

resource "aws_security_group" "cloudwatch-sumologic-lambda-sg" {
  name = "cloudwatch-sumologic-lambda-sg"

  tags {
    Name = "cloudwatch-sumologic-lambda-sg"
  }

  description = "Security group for lambda to move logs from CWL to SumoLogic"
  vpc_id      = "${aws_vpc.dev-vpc.id}"
}

resource "aws_security_group_rule" "https-egress-cloudwatch-sumologic-to-internet" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  security_group_id = "${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_iam_role" "test-cloudwatch-lambda-role" {
  name = "test-cloudwatch-lambda-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "test-cloudwatch-lambda-policy" {
  name = "test-cloudwatch-lambda-policy"
  role = "${aws_iam_role.test-cloudwatch-lambda-role.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole1",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface"
      ],
      "Resource": "*"
    },
    {
      "Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole2",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "arn:aws:ec2:ap-southeast-2:${var.dev_vpc_account_id}:network-interface/*"
    },

    {
      "Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole1",
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:*"
    },
    {
      "Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole2",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
    "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:log-group:/aws/lambda/*"
      ]
    },

    {
      "Sid": "CopiedFromTemplateAWSLambdaAMIExecutionRole",
      "Effect": "Allow",
      "Action": [
         "ec2:DescribeImages"
      ],
      "Resource": "*"
    }


  ]
}
EOF
}

resource "aws_lambda_function" "cloudwatch-sumologic-lambda" {
  function_name    = "cloudwatch-sumologic-lambda"
  filename         = "${var.lambda_dir}/cloudwatchSumologicLambda.zip"
  source_code_hash = "${base64sha256(file("${var.lambda_dir}/cloudwatchSumologicLambda.zip"))}"
  handler          = "cloudwatchSumologic.handler"

  role        = "${aws_iam_role.test-cloudwatch-lambda-role.arn}"
  memory_size = "128"
  runtime     = "nodejs4.3"

  // set low because I'm concerned about cost-blowout in the case of mis-configuration
  timeout = "15"

  vpc_config = {
    subnet_ids         = ["${aws_subnet.dev-private-subnet.id}"]
    security_group_ids = ["${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"]
  }
}

resource "aws_lambda_permission" "test-app-allow-cloudwatch" {
  statement_id  = "test-app-allow-cloudwatch"
  action        = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
  principal     = "logs.ap-southeast-2.amazonaws.com"
  source_arn    = "${aws_cloudwatch_log_group.test-app-loggroup.arn}"
}

resource "aws_cloudwatch_log_subscription_filter" "test-app-cloudwatch-sumologic-lambda-subscription" {
  depends_on      = ["aws_lambda_permission.test-app-allow-cloudwatch"]
  name            = "cloudwatch-sumologic-lambda-subscription"
  log_group_name  = "${aws_cloudwatch_log_group.test-app-loggroup.name}"
  filter_pattern  = ""
  destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}

EDIT: Please note that the above TF code was written years ago, using version 0.11.x - it should still work but there may be better ways of doing things. Specifically, don't use an inline policy like this unless needed, use an aws_iam_policy_document instead - they're just way easier to maintain over time.

Terraform: configuring cloudwatch log subscription delivery to lambda?

2 Answers