Kafka Streams Processor API Usage

0
votes

Can I use processor api in Kafka Streams and read data from a topic in one kafka cluster and pipe data into a kafka topic in ANOTHER kafka cluster?

Is that do-able thing using processor api? I know, its not possible with Stream DSL!

1
apache-kafkaapache-kafka-streams
It could be possible if you manually create a Producer object yourself within the Processor, but as mentioned, MirrorMaker is the official builtin tool for this – OneCricketeer
Duplicate: stackoverflow.com/questions/45847690/… – Matthias J. Sax
If you create your own instance of KafkaProducer you will need to go sync writes to guarantee at-least-once processing. Thus, performance is expected to drop. It's not a recommended pattern. – Matthias J. Sax

1 Answers

3
votes

This is not possible with Kafka Streams. It only works against a single cluster.

To copy data across clusters you need to either use MirrorMaker or one of the available replicator tools, for example a Kafka source connector.