Kinesis stream to AWS lambda

1
votes

I am currently using DynamoDB streams and looking forward to move to Kinesis streams as I want to gain control over the number of records I like to process from the streams.

I have been reading about Kinesis streams and lambda. There are many articles about Kinesis streams and EC2's where multiple consumers and KCL etc..

All I would like to know is, If I go with lambda for Kinesis stream, what is the expected behavior?

  • Like Dynamo streams, when Kinesis has records in the streams, lambda gets triggered with the bunch of records in the event?
  • Whenever there are records in Kinesis stream, Lambda gets triggered and Lambda can go ahead and read the shards to fetch the records and process further.

I am looking for the second option in the above. I do not want the lambda to be triggered with set of records but I want to control the amount I read.

Could anyone explain how can I gain control of Kinesis streams in Lambda?

1
amazon-web-servicesaws-lambdaamazon-kinesis
Here's a sample repository made by terraform. – GNOKOHEAT

1 Answers

0
votes

You can subscribe a lambda to a Kinesis stream. When you subscribe you can specify the batch size, 1 to 1000 as I recall. The lambda will be invoked for each shard in the stream when there are records available up to the batch size. If the lambda errors, (e.g. returns an error or times out) it will be retried with the same data. It will be retried until it succeeds or the records age out of the stream. The lambda will be invoked around once per second per shard if there are records available. The lambda is invoked with whatever records are available up to the batch size.

There will be only one concurrent lambda invocation per shard. This is to guarantee the ordering of records inside a shard.

Generally the behavior for Kinesis streams is the same as for DynamoDB streams.