Flafka (Http -> Flume->Kafka ->Spark Streaming)

Question

I have one use case for real time streaming, we will be using Kafka(0.9) for message buffer and spark streaming(1.6) for stream processing (HDP 2.4). We will receive ~80-90K/Sec event on Http. Can you please suggest a recommended architecture for data ingestion into Kafka topics which will be consumed by spark streaming.

We are considering flafka architecture.

Is Flume listening to Http and sending to Kafka (Flafka )for real time streaming a good option?

Please share other possible approaches if any.

Answer 1

One approach could be Kafka Connect. Look for a source that fit in your needs or develop a custom new one.