Writing data to kinesis stream from hdfs

Question

I have a requirement to write data from hdfs to kinesis stream for further computation. Can some one guide me on how to put data from hdfs to kinesis stream. For example i have parquet format data on hdfs. I want to put data to kinesis stream now. And how can i control the limit up to which is allowed in kinesis.

Thanks.

Answer 1

Streaming data from HDFS to kafka/Kinesis is a moderately common problem. One of the common approach is to use Apache Flume in between of HDFS and Kafka. You need to set Kafka as the destination and HDFS as source.

Flume works with Kinesis. check this. This enables Kinesis to be source or sink.