Why does S3 bucket created in terraform needs bucket policy to grant access to lambda

Question

We use a combination of cloud formation and terraform where some common resources like DynamoDB, S3 are created using terraform and others like APIGateway are created using serverless and cloudformation. All resources are in the same AWS account

I have an S3 bucket in terraform

resource "aws_s3_bucket" "payment_bucket" {
  bucket = "payment-bucket-${var.env_name}"
  acl    = "private"

  tags = merge(
    module.tags.base_tags,
    {
      "Name" = "payment-bucket-${var.env_name}"
    }
  )

  lifecycle {
    ignore_changes = [tags]
  }
}

This creates a private bucket payment-bucket-dev in my AWS account when I run the tf-apply

We have an APIGateway in the same AWS account which is created using serverless and one of the lambda needs accesses to this bucket so I have created an IAM role for the lambda function to grant permission to access the bucket.

  makePayment:
    name: makePayment-${self:provider.stage}
    handler: src/handler/makePayment.default
    events:
      - http:
          path: /payment
          method: post
          private: true
          cors: true
    iamRoleStatementsName: ${self:service}-${self:provider.stage}-makePayment-role
    iamRoleStatements:
      - Effect: Allow
        Action:
          - s3:PutObject
        Resource:
          - arn:aws:s3:::#{AWS::Region}:#{AWS::AccountId}:payment-bucket-${self:provider.stage}/capture/batch/*

But when I run this lambda make-payment-dev , it throws an AccessDenied error unless I add bucket policy granting access to the lambda role

resource "aws_s3_bucket_policy" "payment_service_s3_bucket_policy" { 
..
..
}

Why do I need to add S3 bucket policy when both s3 bucket and the lambda function and role are in the same account? Am I missing something?

Also, If I created the bucket using AWS::S3::Bucket as part of the cloud formation stack the Apigateway is in (we are using serverless), I don't need add bucket policy and it all works fine.

Answer 1

I think the problem is simply that the S3 bucket ARN is incorrect.

S3 bucket ARNs do not have account IDs or regions in them. Use arn:aws:s3:::mybucket/myprefix/*.

Answer 2

The answer depends on what AWS IAM role is applying the terraform plan because the AWS s3 bucket canned ACL rule: "private" restricts bucket access as: Owner gets FULL_CONTROL. No one else has access rights (default). per documentation: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html

You have to be relatively explicit at this point as to who can access the bucket. Often if I'm going with private ACL but want every other role in my AWS account to have access to the bucket I attach a bucket policy to the terraform aws_s3_bucket resource to first allow access to the bucket. Then I explicitly grant the lambda's role access to said bucket via another inline policy.

In your case it would look something like this:

// Allow access to the bucket
data "aws_iam_policy_document" "bucket_policy" {
  statement {
    sid = "S3 bucket policy for account access"

    actions = [
      "s3:ListBucket",
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ]

    principals {
      type = "AWS"

      identifiers = [
        "arn:aws:iam::{your_account_id_here}:root",
      ]
    }

    resources = [
      "arn:aws:s3:::test_bucket_name",
      "arn:aws:s3:::test_bucket_name/*",
    ]

    condition {
      test     = "StringEquals"
      variable = "aws:PrincipalArn"
      values   = ["arn:aws:iam::{your_account_id_here}:role/*"]
    }
  }
}

resource "aws_s3_bucket" "this" {
  bucket = "test_bucket_name"
  acl    = "private"

  policy = data.aws_iam_policy_document.bucket_policy.json
}

// Grant the lambda IAM role permissions to the bucket
data "aws_iam_policy_document" "grant_bucket_access" {
  statement {
    sid = "AccessToTheAppAuxFilesBucket"
    actions = [
      "s3:ListBucket",
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ]

    resources = [
      "arn:aws:s3:::test_bucket_name/*",
      "arn:aws:s3:::test_bucket_name"
    ]
  }
}

// Data call to pull the arn of the lambda's IAM Role
data "aws_iam_role" "cloudformation_provisioned_role" {
  name = "the_name_of_the_lambdas_iam_role"
}

resource "aws_iam_role_policy" "iam_role_inline_policy" {
    name = "s3_bucket_access"
    role = data.aws_iam_role.cloudformation_provisioned_role.arn

    policy = data.aws_iam_policy_document.grant_bucket_access.json
}

Answer 3

It's an open bug. acl and force_destroy aren't well imported with terraform import : https://github.com/hashicorp/terraform-provider-aws/issues/6193