How "Real-Time" DynamoDB stream is?

5
votes

We are experimenting with a new serverless solution where external provider writes to DynamoDB, DynamoDB Stream reacts to a new write event, and triggers AWS Lambda function which propagates changes down the road?

So far it works well, however, sometimes we notice that data is being delayed e.g. no updates would come from Lambda for a few minutes.

After going through a lot of DynamoDB Stream documentation the only term they use is "near real-time stream record" but what is generally "near real-time"? What are the possible delays we are looking at here?

2
amazon-web-servicesamazon-dynamodbreal-timeaws-lambdaamazon-dynamodb-streams
no updates would come from Lambda wouldn't necessarily be the same as the data being delayed. The Lambda function could be the cause of the delay. Have you also verified that the function isn't being invoked previously, possibly throwing errors, and the data you get is the result of Lambda retries? - Michael - sqlbot
@Michael-sqlbot isn't lambda stateless? as I understood new lambda function execution shouldn't have any relation to the previous lambda running. - inside
That's not quite true. Your code should be designed that way, and each container only handles one concurrent invocation, but the containers are often reused... so your function can fail or stall on subsequent invocations if it has design errors triggered by something you didn't clean up. Errors are retried after a delay, also. Read your cloudwatch logs and see if you see any potentially problematic signs of issues. - Michael - sqlbot

2 Answers

4
votes

In my experience, most of the time it is near real-time. However, on a rare occasion you might have to wait a while (in my case, up to half an hour). I assume this was because of hardware or network issues in AWS infrastructure.

3
votes

In most cases, Lambda functions are triggered within half a second after you make an update to a small item in a Streams-enabled DynamoDB table. But event source changes, updates to the Lambda function, changing the Lambda execution role, etc. may introduce additional latency when the Lambda function is run for the first-time.