Multiple AWS Lambda functions on a Single DynamoDB Stream

12
votes

I have a Lambda function to which multiple DynamoDB streams are configured as event sources and this is a part of a bigger pipeline. While doing my checks, I found some missing data in one of the downstream components. I want to write a simpler Lambda function which is configured as an event source to one of the earlier mentioned DynamoDB streams. This would cause one of my DynamoDB streams to have two Lambda functions reading from it. I was wondering if this is OK? Are both Lamdba functions guaranteed to receive all records placed in the stream and are there any resource (Read/Write throughput) limits I need to be aware of. Couldn't find any relevant documentation for this on the AWS website, but I did find this regarding processing of shards

To access a stream and process the stream records within, you must do the following:

  • Determine the unique Amazon Resource Name (ARN) of the stream that you want to access.
  • Determine which shard(s) in the stream contain the stream records that you are interested in.
  • Access the shard(s) and retrieve the stream records that you want.

Note No more than 2 processes at most should be reading from the same Streams shard at the same time. Having more than 2 readers per shard may result in throttling.

Not sure how the above relates to cases where Streams are configured as Event sources to Lambdas as opposed to manually reading from a Stream using the API.

2
amazon-web-servicesaws-lambdaamazon-dynamodb-streams
when you refer to two lambdas here do you mean more than one instance of the same lambda function or are the two lambda functions doing different things? – shrewquest

2 Answers

12
votes

You can have multiple Lambdas using the same stream as an event source. They will not interfere with each other. But as the documentation says: "Note No more than 2 processes at most should be reading from the same Streams shard at the same time. Having more than 2 readers per shard may result in throttling." So if you heavily utilize your streams you should not have more than two Lambdas connected to them.

7
votes

This AWS Blog post https://aws.amazon.com/de/blogs/database/how-to-perform-ordered-data-replication-between-applications-by-using-amazon-dynamodb-streams/ suggest that you attach only one Lambda to the DDB stream and use a fan out pattern for parallel processing. This will help you processing the DDB items in order.