Set up S3 Bucket level Events using AWS CloudFormation

Question

I am trying to get AWS CloudFormation to create a template that will allow me to attach an event to an existing S3 Bucket that will trigger a Lambda Function whenever a new file is put into a specific directory within the bucket. I am using the following YAML as a base for the CloudFormation template but cannot get it working.

---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  SETRULE:
    Type: AWS::S3::Bucket
      Properties:
        BucketName: bucket-name
        NotificationConfiguration:
          LambdaConfigurations: 
            - Event: s3:ObjectCreated:Put
              Filter: 
                S3Key:
                  Rules:
                    - Name: prefix
                      Value: directory/in/bucket
              Function: arn:aws:lambda:us-east-1:XXXXXXXXXX:function:lambda-function-trigger
              Input: '{ CONFIGS_INPUT }'

I have tried rewriting this template a number of different ways to no success.

have you added the permissions for lambda invocation on your S3 bucket? — Juned Ahsan
I have gotten a few different errors depending on the slight alterations I have been making, most recently I got a YAML not well-formatted error on the line that just reads Properties:. And the lambda function is defined normally under the lambda functions service in AWS. — mattc-7

Matus Dubrava Matus Dubrava · Accepted Answer · 2019-11-07T11:07:48

Since you have mentioned that those buckets already exists, this is not going to work. You can use CloudFormation in this way but only to create a new bucket, not to modify existing bucket if that bucket was not created via that template in the first place.

If you don't want to recreate your infrastructure, it might be easier to just use some script that will subscribe lambda function to each of the buckets. As long as you have a list of buckets and the lambda function, you are ready to go.

Here is a script in Python3. Assuming that we have:

2 buckets called test-bucket-jkg2 and test-bucket-x1gf
lambda function with arn: arn:aws:lambda:us-east-1:605189564693:function:my_func

There are 2 steps to make this work. First, you need to add function policy that will allow s3 service to execute that function. Second, you will loop through the buckets one by one, subscribing lambda function to each one of them.

import boto3

s3_client = boto3.client("s3") 
lambda_client = boto3.client('lambda')

buckets = ["test-bucket-jkg2", "test-bucket-x1gf"]
lambda_function_arn = "arn:aws:lambda:us-east-1:605189564693:function:my_func"

# create a function policy that will permit s3 service to 
# execute this lambda function

# note that you should specify SourceAccount and SourceArn to limit who (which account/bucket) can
# execute this function - you will need to loop through the buckets to achieve 
# this, at least you should specify SourceAccount
try:
    response = lambda_client.add_permission(
        FunctionName=lambda_function_arn,
        StatementId="allow s3 to execute this function",
        Action='lambda:InvokeFunction',
        Principal='s3.amazonaws.com'
        # SourceAccount="your account",
        # SourceArn="bucket's arn"
    )
    print(response)
except Exception as e:
    print(e)

# loop through all buckets and subscribe lambda function 
# to each one of them
for bucket in buckets:
    print("putting config to bucket: ", bucket)
    try:
        response = s3_client.put_bucket_notification_configuration(
            Bucket=bucket,
            NotificationConfiguration={
                'LambdaFunctionConfigurations': [
                    {
                        'LambdaFunctionArn': lambda_function_arn,
                        'Events': [
                            's3:ObjectCreated:*'
                        ]
                    }
                ]
            }
        )
        print(response)
    except Exception as e:
        print(e)

Set up S3 Bucket level Events using AWS CloudFormation

2 Answers