I am using a function created in AWS Lambda to trigger and process every record inserted into a DynamoDB table.
But I recently noticed that a lot of the records were missing and not being processed. For example, for every 50,000 records inserted, maybe around 2000-3000 will get processed. :(
I also noticed that this number being processed slowly increases by around 100-200 records every 5 minutes or so for some time. I checked this using the aws command line:
aws dynamodb scan --table-name myDynamoTable --select "COUNT"
I am using the default settings to create the Lambda function if that helps.
Is this a problem with Lambda function or the configuration of MY Lambda function?
Are there any properties/settings that I can change to improve its functionality?
Finally, if the worst comes to the worst, what can I use as an alternative (reliable one) for the Lambda function? Can I build one on my own?
TIA.
0
votes
Can you share the dynamodb stream configuration? Are you sure all the items are new inserts? Is it possible for you to delete all the items and insert 100 items and see what happens? Need to debug whether it is a stream or lambda problem? Did you insert all 5k items at the same time?
– notionquest
@notionquest How do I find the stream configuration? Clicking the "Manage Streams" button and enabling it is the only experience I have with DynamoDB streams. :) I will check and see how it behaves when I delete and insert items like you suggested. Do you mean inserting 5k items simultaneously or sequentially in quick succession?
– user6732360
Try inserting sequentially and check the result. Some 100 to 500 items should be sufficient.
– notionquest
@notionquest I tried 500 inserts, 500 updates and 500 deletes to the table. All the operations were being recorded and processed by the lambda function. I then tried 10000 inserts, 10000 updates and 10000 deletes. All the insert and delete operations were being processed, but many of the updates were missing. I sometimes got a count of only 3000 processed updates when in fact 10000 were done. The lambda function seems to be not working when the number of records being updated are huge. Is this a possible limitation of Lambda or could it be the fault of my configuration and/or code?
– user6732360
