Kinesis stream is used to feed messages from calling application and we take the message from Kinesis stream and process. KPL (Onpremise) is used to produce (feed) the data into Kinesis and KCL (@ AWS EC2) is used at consumer end KPL is producing the messages at good rate but consumer is taking more time due to processing time Question1>>How we can improve consumer rate of consumption? we assume consumer will read messages concurrently (concurrency is proportional to shard count) from stream Question2>>We need to have auto scaling of shards based on consumer rate, what is metric is advisable for scaling based on consumption?
1 Answers
0
votes
Kinesis consumers should be very fast at consuming data as if you are using the KCL correctly it in fact batches the data that it forwards onto the consumer processes. To answer your questions:
- Make sure you are using the "push model" on your consumers. This means AWS will take care of batching as much records as possible to reduce network IO and allow fast processing. In my applications it can batch 700 records at a time thus ensuring rapid processing. See link here with details of push/pull model.
- There are ad-hoc solutions for auto-scaling shards (not sure you can auto-scale now direct with kinesis) however you would want to monitor metrics such as MillsBehindLatest for every shard and this would be on the consumer end. However this gets tricky as you need to check every shard on the stream. Also you would need to monitor the producer side also
My view keep it simple, just provision more than enough capacity as Kinesis is able to handle lots of data on even just a few shards and provision a couple of instances to process the data in different availability zones. Increase shards/instances as data adjusts over time.
