Spring Cloud Data Flow (SCDF) + Apache Kafka - Where to Deploy the Kafka Brokers?

1
votes

I am not sure if I understand the relation between Spring Cloud Data Flow (SCDF) and Apache Kafka Brokers (and its related Zookeeper).

I read several docs and watched a webinar about Spring Cloud Data Flow (SCDF). I understand the different binding options (RabbitMQ, Kafka, ...) for communication between different microservices in this framework.

I find a lot of information about how to use Kafka, but not where to run / operate it. Is my assumption correct that you have to install / manage / operate Kafka brokers (+ Zookeeper) independently of SCDF? SCDF "just" uses Kafka brokers for the internal communication and - optionally - for business logic within the microservices (e.g. by using a Kafka Producer or Consumer API, right?

Even if I use a framework like Mesos or Kubernetes for SCDF, the Kafka Brokers (+ ZK) are still run on their own, either also on Mesos or Kubernetes or somewhere else.

SCDF can be used to deploy Kafka applications which communicate with Kafka brokers - this can be Kafka consumers / producers, Kafka Streams apps or even other parts of the Kafka ecosystem like Kafka Connect or KSQL. All of these could run within a Spring Boot application / microservice in SCDF.

Are these statements correct, or do I miss something?

2
javaapache-kafkaspring-cloudspring-cloud-streamspring-cloud-dataflow

2 Answers

3
votes

SCDF "just" uses Kafka brokers for the internal communication and - optionally - for business logic within the microservices (e.g. by using a Kafka Producer or Consumer API, right?

To be clear, SCDF does not use Kafka or any other brokers internally. It is the apps themselves that rely on it at runtime. The apps specifically are based on Spring Cloud Stream programming model. SCDF simply orchestrates such apps into a fully coherent data pipeline. You can read more details about the programming model and how they interact with the binder implementation (brokers) from here and here respectively.

Even if I use a framework like Mesos or Kubernetes for SCDF, the Kafka Brokers (+ ZK) are still run on their own, either also on Mesos or Kubernetes or somewhere else.

Yes - there is no colocation of any of these individual systems. They are expected to be run and operationalized separately.

1
votes

Yes you need to manage the brokers, dataflow won't manage or deploy your middleware infrastructure, such as message brokers or databases