What are the differences between Apache Beam and Apache Kafka with respect to Stream processing? I am trying to grasp the technical and programmatic differences as well.
Please help me understand by reporting from your experience.
10
votes
Beam requires a cluster scheduler to run the code. Kafka Streams can be embedded within any Java application. That's one of the main differences. Beam can communicate with more streams than only Kafka
– OneCricketeer
Cluster scheduler meaning "Runners" right? Beam stream cannot be embedded within any java app? How do we find Beam can communicate with more streams than Kafka?
– Stella
I don't know Beam terminology. AFAIK, you cannot run Beam in a standalone Java application. It would need ran within a scheduler like YARN or Mesos. And Beam can read from Google DataFlow, for example, Kafka Streams cannot.
– OneCricketeer
2 Answers
14
votes
Beam is an API that uses an underlying stream processing engine like Flink, Storm, etc... in one unified way.
Kafka is mainly an integration platform that offers a messaging system based on topics that standalone applications use to communicate with each other.
On top of this messaging system (and the Producer/Consummer API), Kafka offers an API to perform stream processing using messages as data and topics as input or output. Kafka Stream processing applications are standalone Java applications and act as regular Kafka Consummer and Producer (this is important to understand how these applications are managed and how workload is shared among stream processing application instances).
Shortly said, Kafka Stream processing applications are standalone Java applications that run outside the Kafka Cluster, feed from the Kafka Cluster and export results to the Kafka Cluster. With other stream processing platforms, stream processing applications run inside the cluster engine (and are managed by this engine), feed from somewhere else and export results to somewhere else.
One big difference between Kafka and Beam Stream API is that Beam makes the difference between bounded and unbounded data inside the data stream whereas Kafka does not make that difference. Thereby, handling bounded data with Kafka API has to be done manually using timed/sessionized windows to gather data.
9
votes
Beam is a programming API but not a system or library you can use. There are multiple Beam runners available that implement the Beam API.
Kafka is a stream processing platform and ships with Kafka Streams (aka Streams API), a Java stream processing library that is build to read data from Kafka topics and write results back to Kafka topics.
