Amazon Kinesis dynamically stream resize

Question

I am working on Amazon Kinesis Api and Kinesis Client library, I have created one producer to put data into stream and have multiple consumer applications to read data from that stream.

I have scenario to increase and decrease size of stream dynamically as per input stream size and output stream size and also using count of Consumer application.

I found some useful source to count number of shard from amazon website but don't get how to calculate . Source URL is : http://docs.aws.amazon.com/kinesis/latest/dev/how-do-i-size-a-stream.html

Need some understanding on this.

Thanks

Answer 1

AWS support suggests looking at the following open source project. It was created by one of their solution architects.

https://github.com/awslabs/amazon-kinesis-scaling-utils

It can be run manually (cli) or automatically (deployed WAR) to scale up/down with your application.

Answer 2

You could take a look at Themis, a framework that supports autoscaling of Kinesis streams, developed at Atlassian. The tool is very easy to configure, comes with a Web UI, and supports different autoscaling modes (e.g., proactive and reactive autoscaling).

(Apologies for posting in an old thread, but the answer may still be interesting for readers discovering this thread.)

Answer 3

You can dynamically resize stream with using Amazon Cloud Watch service, you just create Alarms based on stream using different metrics like put.byteRecords and get.byteRecords and detect alarm state.

After that based on those alarm state as "ALARM", increase capacity of your stream using resharding, you can do same scenario to decrease capacity of your stream.

for more information visit this link : http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-using-api-java.html

Answer 4

Since November 2016, you can easily scale your Amazon Kinesis streams using the updateShardCount function, Lambda functions and Amazon Cloud Watch Alarms.

You may find this post really useful.

Answer 5

I have created npm module which helps in auto scaling kinesis stream.

You can find detailed information at Amazon Kinesis Scaling

This is npm module which scale amazon kinesis as per current traffic needs. This module continuously monitor traffic in kinesis stream and split and merge shards as needed. E.g. if your application needs to handle 5000 req/sec then you need to have 5 shards. Since traffic on your application can varies a lot so does number of shards. If your application needs to handle 20000 req/sec at peak time then you need to have 20 shards but when at other time you may required only 5 shards.