Confluent - Splitting Avro messages from one kafka topic into multiple kafka topics

1
votes

We have an incoming kafka topic with multiple Avro schema based messages serialized into it.

We need to split the messages in Avro format into multiple other kafka topics based on certain value of a common schema attribute.

                             |------> [OUTGOING TOPIC(AVRO) - A] 
[INCOMING TOPIC(AVRO)] ----->|------> [OUTGOING TOPIC(AVRO) - B]
                             |------> [OUTGOING TOPIC(AVRO) - C]

Would like to understand how to achieve it while avoid building an intermediate client to do this splitting/routing in confluent platform.

I explored the kafka connector but did not find an existing connector which does this work.

1
apache-kafkaavroapache-kafka-streamsapache-kafka-connectconfluent-platform
You can either use KSQL or Kafka Streams. – Giorgos Myrianthous
KSQL might not work, because the input topic does not have one schema. – Matthias J. Sax

1 Answers

1
votes

You can write a Kafka Streams application and use branch():

KStream input = builder.stream("topic");
KStream[] splitStreams = input.branch(...);
splitStream[0].to("output-topic-1");
splitStream[1].to("output-topic-2");
// etc.