Kafka --> spring cloud Data Flow(RabbitMQ) --> Kafka

0
votes

The problem I have is moving data with spring cloud data flow (SCDF) through an environment to get data to the right place. the following is the situation:

  • I have a productionized version of Kafka which I am only allowed to subscribe to (read Only). (3 node cluster: node1:9092,node2:9092,node3:9092)
  • I have a instance of SCDF running in a Kubernetes cluster that uses a RabbitMQ backbone (all on one server)
  • I have a single Node Kafka server that is being used for another company to get data (ssl protected)

When looking at the SCDF I am not seeing a way to define Kafka as a source and read to a processor (gonna be custom) and write to a different Kafka instance.

1
apache-kafkaspring-cloud-streamspring-cloud-dataflow

1 Answers

0
votes

You can use a named destination together with multi-binder support.

RabbitMQ Binding -> processor -> named Kafka Destination

https://dataflow.spring.io/docs/feature-guides/streams/taps/#tapping-a-stream

In Spring Cloud Stream terms, a named destination is a specific destination name in the messaging middleware or the streaming platform. It could be an exchange in RabbitMQ or a topic in Apache Kafka. In Spring Cloud Data Flow, the named destination can be treated as either a direct source or sink, based on whether it acts as a publisher or a consumer. You can either consume data from a destination (example: a Kafka topic) or you can produce data for a destination (for example, a Kafka topic). Spring Cloud Data Flow lets you build an event-streaming pipeline from and to the destination on the messaging middleware by using the named destination support.

http > :user-click-events

:user-click-events > jdbc

https://docs.spring.io/spring-cloud-stream/docs/3.1.3/reference/html/spring-cloud-stream.html#multiple-binders

When multiple binders are present on the classpath, the application must indicate which binder is to be used for each destination binding.

So:

:RO_Kafka > processor_1 | ... | processor_n > :otherKafka

With processor_1 and processor_n configured with multi-binders.