Kafka + Flume or Kafka + Storm/Spark streaming

Question

Till now my architecture includes Storm/Spark streaming for data pre-processing but I have read somewhere that integration of Flume with Kafka has many benefits. I am doing real time streaming of 5 sources including sensor, video, structured, telematics and social media data.

Can anybody let me know why one should use kafka and flume together?

Answer 1

About Flume

• You can use flume to ensure your message can easily collect(won`t lost,when you collect metadata for further usage)

About Kafka（Kafka introduction）

• You can use kafka to ensure your message correct,and easily to consume(won`t lost or replicate,when you want to calculate)

Summary

In factly,flume is create for collection as data collector,Kafka is create for store and consume data as dispatcher.Kafka is the message queue,just like redis.

Of course,If you want,you can use Flume alone without Kafka.