When deploying Spring-Cloud-Data-Flow on AKS, how do I specify using externally hosted Kafka cluster

Question

I've tried a helm based deployment (https://docs.spring.io/spring-cloud-dataflow-server-kubernetes/docs/current-SNAPSHOT/reference/htmlsingle/#_helm_installation) of SCDF and this can either use a RabbitMQ or Kafka based messaging system - both of which are options to be deployed within the Helm chart. However, my project already has its own Kafka instance running on the Confluent cloud that I'd like to use instead of the "internal" instance. I have also then tried the Kubectl deployment of SCDF (https://docs.spring.io/spring-cloud-dataflow-server-kubernetes/docs/current-SNAPSHOT/reference/htmlsingle/#_deploying_with_code_kubectl_code) and see that I needed to run up one of the two messaging components again. I'd like to know the best way to configure SCDF to use my own Kafka instance - can anyone shed any light one this please?

Ilayaperumal Gopinathan Ilayaperumal Gopinathan · Accepted Answer · 2020-07-14T16:32:04

The document references you have above are pretty old and hence, you may want to update to the latest one.

For the latest on SCDF, you need to refer here Also, there is a SCDF microsite you can refer as well.

To answer your specific question, since Apache Kafka is used as the binder in your streaming application, you have two options to set the external Kafka cluster configuration:

Set via Skipper server configuration as a server level property (Since Skipper server is responsible for continuous deployment of streaming applications in SCDF, you need to set the stream app related configurations in Skipper server config).

An example of this configuration can be found here. You will notice how environmentVariables K8s deployer property is used to configure Kafka specific properties.

data:
  application.yaml: |-
    spring:
      cloud:
        skipper:
          server:
            platform:
              kubernetes:
                accounts:
                  default:
                    environmentVariables: 'SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=${KAFKA_SERVICE_HOST}:${KAFKA_SERVICE_PORT},SPRING_CLOUD_STREAM_KAFKA_BINDER_ZK_NODES=${KAFKA_ZK_SERVICE_HOST}:${KAFKA_ZK_SERVICE_PORT}'

The other option is to set the Kafka specific properties as app deployment properties at the time of stream deployment. This is specifically useful when you want to override the Kafka configuration during stream deployment.

If you have a same Kafka configuration for all the streams, then you can choose the first option and set the Skipper ConfigMap to specify Kafka configuration as environmentVariables.

When deploying Spring-Cloud-Data-Flow on AKS, how do I specify using externally hosted Kafka cluster

1 Answers