Advise on cross account s3 and lambda integration

0
votes

There are two accounts:

Account-A (our team does not own this service) -> s3 Bucket (exampleBucket) is located.

Account-B (our team owns) -> Lambda function (myLambda)

We need to execute a Lambda function (myLambda) whenever a new object is created in the s3 bucket (exampleBucket).

I have found at-least 2 ways to do this:

S3 (Event Notification) -> Lambda

S3 (Event Notification) -> SNS Topic -> Lambda reads the notification from SNS

S3 (Event Notification) -> SQS Topic -> Lambda reads the notification from SQS

Additional Info:

  1. Both Lambda and S3 are in the same region (different AWS accounts)

  2. Assume we have all the necessary permissions

  3. We get a new object in the bucket once every (~5 seconds)

  4. There are enough Lambdas (concurrent executions) to handle all the invocations.

I need how to determine which would be better, in terms of:

  1. Latency (Time of creation of object in S3 - Executing Lambda function)

  2. Maintenance

  3. Monitoring

  4. Scaling (I am assuming since all the services are managed by AWS, this shouldn't be an issue)

1
amazon-web-servicesamazon-s3aws-lambdanotificationsamazon-sns
Are you saying that you have the cross-account trigger of the AWS Lambda function working? That is the hardest part! Clearly going direct to Lambda would be the lowest-latency, but if you are expecting things to go wrong you might appreciate having a Dead Letter Queue to collect failed Lambda executions using SQS. - John Rotenstein
All the permissions are enabled, I am trying to understand which approach would be the best for the notification system. - mvsnbharath

1 Answers

1
votes

I would consider what kind of workloads you are looking at and what possible future use-cases may look like.

  1. Do you expect that the Lambda invocations often may fail? If so, is it important for you to be able to retry processing the events? If that's the case, SQS or SNS with DLQ configured would be good options.
  2. Do you expect that you will want to hook up additional Lambdas or other downstream actions based on these events in the future? If so, you may want to use SNS to be able to add multiple subscriptions.
  3. How do you plan to deploy updates to the Lambda? If you occasionally want to be able to take it offline to make changes, having DLQs available to pick up undeliverable notifications can help.

Latency is unlikely to be an issue as you're in the same region.

Monitoring can also easily be set up with either SNS or SQS. You can set up CloudWatch alarms based on failed deliveries for SNS, number of messages in the DLQ, etc.