Terraform cloudwatch retention logic

Question

I'm trying to add logic to my terraform script so that when a lambda is created a CW retention logic is added, so our logs clear after 30 days. What I'm seeing is that when terraform is run to update an existing lambda (that was deployed prior to my new retention logic being added) my job fails with the below error.

• aws_cloudwatch_log_group.lambda-deploy: 1 error(s) occurred:

• aws_cloudwatch_log_group.lambda-deploy: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists status code: 400, request id: e500eb50-4a81-11e9-9c08-7152b4a0ad31: The CloudWatch Log Group '/aws/lambda/{lambda-name}' already exists.

Below is how I have my terraform code set up:

resource "aws_lambda_function" "lambda-deploy" {
  filename      = "${var.filename}"
  function_name = "${var.functionname}"
  role          = "${var.role}"
  handler       = "${var.handler}"
  runtime       = "${var.runtime}"
  publish       = "${var.publish}"
  memory_size   = "${var.memory_size}"
  timeout       = "${var.timeout}"
  description   = "${var.description}"

  layers = "${var.layers}"

  environment {
    variables = "${var.envVars}"
  }

  tags {
    PLATFORM        = "${var.tag_PLATFORM}"
    BUSINESS_UNIT   = "${var.tag_BUSINESS_UNIT}"
    CLIENT          = "${var.tag_CLIENT}"
    BUSINESS_REGION = "${var.tag_BUSINESS_REGION}"
  }

  vpc_config {
    subnet_ids         = "${var.subnet_ids}"
    security_group_ids = "${var.security_group_ids}"
  }
}

#Below logic will add cloud watch retention logic so logs rotate after 30 days.
resource "aws_cloudwatch_log_group" "lambda-deploy" {
  name              = "/aws/lambda/${aws_lambda_function.lambda-deploy.function_name}"
  retention_in_days = "30"
}

My question is, is it possible for the aws_cloudwatch_log_group resource to check if a cloudwatch group has been created and just update the retention policy instead of trying to create the Log Group?

Answer 1

1. Comment out the name argument

The above example, for instance:

resource "aws_cloudwatch_log_group" "lambda-deploy" {
  name = "/aws/lambda/${aws_lambda_function.lambda-deploy.function_name}"
  retention_in_days = "30"
}  

Becomes:

resource "aws_cloudwatch_log_group" "lambda-deploy" {
  #name = "/aws/lambda/${aws_lambda_function.lambda-deploy.function_name}"
  retention_in_days = "30"
}

2. Import the resource

   terraform import aws_cloudwatch_log_group.lambda-deploy /aws/lambda/${aws_lambda_function.lambda-deploy.function_name}

Where ${aws_lambda_function.lambda-deploy.function_name} is what you have in AWS.

3. Uncomment your 'name' argument and run terraform plan, your log group is now managed by terraform...phew!

Answer 2

Since you don't show us the iam role policy for lambda function, I have to guess.

My feeling is the iam role and policy for lambda has the permission logs:CreateLogGroup. So lambda function will take care of creating log group if not exist.

When you update, this log group is not managed by terraform, it will report the issue.

Could you remove the permission logs:CreateLogGroup and remove the log group, then try again?

Answer 3

It is TECHNICALLY possible for it to detect if it exists and optionally not create it. Hypothetically, you could make a data block to import a pre-existing the log group and then a resource with a dynamic count to turn it on or off creation. But you wouldn't want to do it that way, that wierd hack would be in your infra code forever. You want to keep your terraform clean and geared toward recreating your infrastructure from scratch.

Assuming you already had a lambda, that lambda with AWSLambdaBasicExecutionRole made it's own log group outside terraform with no retention settings, and now you want to add retention settings control to terraform, you have a bit of a pickle on your hands. Terraform will want to create that log group because it doesn't know it exists, but can't because the name is in use.

So you have to import the existing log group to map to the resource declaration you made in terraform. Then when you apply, terraform will adjust the existing log group to your TF configuration.

This morning I was in this exact situation, my lambdas (and other things) are created in a module thats are loaded in about 10 different stacks (in 3 environments) and I didn't want to do this fully by hand, so I will share this little set of bash commands I put together to make it easier. This is hacky and depends on the (ever changing) terraform console output but for me today on Terraform v0.14.4, the output of an apply would fail like this:

Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.

Enter a value: yes

module.api_gateway.aws_cloudwatch_log_group.this: Creating...

Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists: The CloudWatch Log Group 'API-Gateway-Execution-Logs_7pnv677kwa/0' already exists.

The following commands would capture that apply output, parse the resource namespace and aws name out of the last two lines, and then do an import. It's not an optimised script, just something I threw together to get a job done.

terraform apply 2>&1 | tee out.txt
resource=$(cat out.txt | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g" | grep Creating | grep cloudwatch | cut -f 1 -d:)
name=$(cat out.txt | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g" | grep Creating | grep exists | cut -f 2 -d\')
terraform import $resource $name
terraform apply
rm out.txt