Terraform - Error putting S3 notification configuration: InvalidArgument: Unable to validate the following destination configurations

Question

I'm trying to configure a Lambda event notification in S3 using terraform v0.11.8. This is how my terraform looks like -

###########################################
#### S3 bucket 
###########################################
resource aws_s3_bucket ledger_summary_backups {
  bucket = "${var.environment_id}-ledgersummary-backups"
  acl    = "private"
  tags   = local.common_tags
}

###########################################
######  Lambda Functions
###########################################
resource aws_s3_bucket_notification bucket_notification {
  bucket = aws_s3_bucket.ledger_summary_backups.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.account_restore_ledgersummary_from_s3.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "AWSDynamoDB/"
    filter_suffix       = ".gz"
  }

  depends_on = [aws_lambda_permission.allow_bucket]
}

resource aws_lambda_permission allow_bucket {
  statement_id  = "AllowS3Invoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.account_restore_ledgersummary_from_s3.arn
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.ledger_summary_backups.arn
}

resource aws_lambda_function account_restore_ledgersummary_from_s3 {
  function_name    = "${var.environment_id}-AccountService-${var.account_ledgersummary_restore_event_handler["namespace"]}"
  description      = "Event Handler for ${var.account_ledgersummary_restore_event_handler["name"]}"
  runtime          = "python3.7"
  memory_size      = 256
  handler          = "RestoreDynamoDbFromS3.lambda_handler"
  role             = aws_iam_role.account_s3_to_dynamodb_lambda_role.arn
  timeout          = var.account_ledgersummary_restore_event_handler["lambda_timeout"]
  filename         = data.archive_file.RestoreDynamoDbFromS3.output_path
  source_code_hash = filebase64sha256(data.archive_file.RestoreDynamoDbFromS3.output_path)

  vpc_config {
    security_group_ids = slice(list(aws_security_group.inbound_core_security_group.id, data.terraform_remote_state.environment_state.outputs.default_vpc_security_group), local.sg_list_start, 2)
    subnet_ids         = data.terraform_remote_state.environment_state.outputs.private_subnets
  }

  environment {
    variables = {
      ENVIRONMENT = var.environment_id
    }
  }

The IAM role I've attached to the lambda function has AmazonS3FullAccess and AWSOpsWorksCloudWatchLogs policies attached. I'm able to add the event in AWS Console but in terraform it's throwing the below error

2021-04-08T18:57:23.6474244Z ##[error][1m[31mError: [0m[0m[1mError putting S3 notification configuration: InvalidArgument: Unable to validate the following destination configurations
2021-04-08T18:57:23.6475638Z ##[error]  status code: 400, request id: 3Y8F88E77CX8NZ2N, host id: q88f+go45dalh7+eiYSErkkeDbI0nv+9j7AAecvBWSJoBjZc8hvh2LVeaqo5aGIJv4+aoKwUlgk=[0m
2021-04-08T18:57:23.6476912Z ##[error][0m  on dynamodb-upgrade.tf line 150, in resource "aws_s3_bucket_notification" "bucket_notification":
2021-04-08T18:57:23.6478084Z ##[error] 150: resource aws_s3_bucket_notification bucket_notification [4m{[0m
2021-04-08T18:57:23.6478895Z ##[error][0m
2021-04-08T18:57:23.6479554Z ##[error][0m[0m
2021-04-08T18:57:23.7908949Z ##[error]Failed to apply changes to configuration for workspace mahbis01: Cake.Core.CakeException: Terraform: Process returned an error (exit code 1).
2021-04-08T18:57:23.7911412Z ##[error]   at Cake.Core.Tooling.Tool`1.ProcessExitCode(Int32 exitCode)
2021-04-08T18:57:23.7913466Z ##[error]   at Cake.Core.Tooling.Tool`1.Run(TSettings settings, ProcessArgumentBuilder arguments, ProcessSettings processSettings, Action`1 postAction)
2021-04-08T18:57:23.7915512Z ##[error]   at Cake.Terraform.TerraformApplyRunner.Run(TerraformApplySettings settings)
2021-04-08T18:57:23.7917197Z ##[error]   at Submission#0.ApplyConfiguration(String env)
2021-04-08T18:57:23.7924027Z ##[error]An error occurred when executing task 'Deploy'.
2021-04-08T18:57:23.7974563Z ##[error]Error: One or more errors occurred.
2021-04-08T18:57:23.7976420Z ##[error]  Terraform: Process returned an error (exit code 1).
2021-04-08T18:57:23.8371520Z ##[error]System.Exception: Unexpected exit code 1 returned from tool Cake.exe
2021-04-08T18:57:23.8372857Z    at Microsoft.TeamFoundation.DistributedTask.Task.Internal.InvokeToolCmdlet.ProcessRecord()
2021-04-08T18:57:23.8373538Z    at System.Management.Automation.CommandProcessor.ProcessRecord()
2021-04-08T18:57:23.8586136Z ##[error]PowerShell script completed with 1 errors.

What am I missing in terraform?

Answer - I added bucket policy to my s3 bucket and added lambda function dependency in bucket notification

resource aws_s3_bucket ledger_summary_backups {
  bucket = "${var.environment_id}-ledgersummary-backups"
  acl    = "private"

  policy = <<EOF
{
      "Version": "2012-10-17",
      "Statement": [
         {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::${var.environment_id}-agl-event-files/*"
        }
      ]
    }
EOF

  tags   = local.common_tags
}

resource aws_s3_bucket_notification bucket_notification {
  bucket = aws_s3_bucket.ledger_summary_backups.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.account_restore_ledgersummary_from_s3.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "AWSDynamoDB/"
    filter_suffix       = ".gz"
  }

  depends_on = [
    aws_lambda_permission.allow_bucket,
    aws_lambda_function.account_restore_ledgersummary_from_s3
    ]
}

Serkan GÜMÜŞ Serkan GÜMÜŞ · Accepted Answer · 2021-08-31T13:29:30

There is a conflict between s3 notification and lambda permission. Even though I put depends_on within s3 notification for lambda_permission, I got the same error. So, I've solved this problem to add null_resource like this. it waits a little bit right after the creation of lambda permission and creates bucket notification.

resource "null_resource" "wait_for_lambda_trigger" {
  depends_on   = [aws_lambda_permission.s3_trigger]
  provisioner "local-exec" {
    command = "sleep 3m"
  }
}

resource "aws_s3_bucket_notification" "bucket_create_notification" {
  bucket = aws_s3_bucket.aws_capstone_bucket.id
  depends_on   = [null_resource.wait_for_lambda_trigger]
  
  lambda_function {
    lambda_function_arn = aws_lambda_function.s3_to_dynamo_Lambda.arn
    events              = ["s3:ObjectCreated:*", "s3:ObjectRemoved:*"]
    filter_prefix       = "media/"
  }
}

Terraform - Error putting S3 notification configuration: InvalidArgument: Unable to validate the following destination configurations

2 Answers