I have two types of events that I plan to put in kinesis. Event A has high frequency so I need the batch-size as 100 and maxRecordAge as 1 minute. Event B is less often and I would like to keep the batch-size as 1 as I want to process event as soon as possible. I plan to keep both the events in same kinesis but on different shards. Is there a way to configure batch-size based on shards? I am using Kinesis lambda trigger to trigger the lambda.
1 Answers
1
votes
No, there is no way to define the batch size per shard. Batchsize configuration is defined on the stream level.
There are two possibilties to handle this scenario:
Either you have two lambda functions as acting as consumers, each one with different batch size, and based on the message key, you can filter the messages for high frequency/low frequency data
Split the data into two seperate streams and add a lambda consumer to each stream.
