AWS Lambda cross-region output processing

Question

I have a lambda function deployed in 3 AWS regions. I want to invoke these functions which will write a output file in S3 within those region buckets.

Then I want to collect these files (from 3 region buckets) to do further processing on them.

What would be the best way to do that using AWS service.

I have explored below :

[1] use SNS to trigger the cross-region lambda jobs. But not sure how to get all the 3 output files from the region buckets to do further processing.

[2] use AWS Simple Workflow but again you will have to create it in 3 regions and cross-region activity is not supported.

Answer 1

You're going to want to setup an event on your S3 bucket to kick off a Lambda whenever an item is written to it. From:

http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

For your use-case, the lower-scale and simpler option is to trigger on a putObject to a S3 bucket, it publishes an event to SNS which will directly initiate a Lambda via a CloudWatch Event.

For larger scale, you'll want to publish your event to an SQS queue which you can now do directly seen here.

Also of-note: From your question I don't think you know that lambdas can read/write to ANY region of S3. They do not need to be IN that region. But they will read/write faster to assets in that region, and won't incur bandwidth fees