User topic management using Kafka Stream Processor API

0
votes

I have just started my hands dirty with kafka. I have gone through this. It only says data/topic management for kafka stream DSL. Can anyone share any link for same sort of data management for Processor API of kafka stream? I am specially interested for user and internal topic management of Processor API.

TopologyBuilder builder = new TopologyBuilder();

// add the source processor node that takes Kafka topic "source-topic" as input
builder.addSource("Source", "source-topic")

From where to populate this source topic with input data before stream processor starts consuming the same?

In short, can we write to kafka "Source" topic using streams, like as producer writes to topic? Or is stream only for parallel consumption of topic? I believe we should as "Kafka's Streams API is built on top of Kafka's producer and consumer clients".

2
apache-kafkaapache-kafka-streams
Can you explain what you want to achieve exactly? – subzero
@Subhransu: I want to put each row of input excel data as single message of a user defined topic and then using stream process it for business logic till the sink. I am having the data to put in source topic, but no clue how to do it programmatically. – sumit
I guess I found my answer here in this link – sumit
I dont understand how that solution points to writing a kafka topic from an excel file. – subzero

2 Answers

0
votes

Yes, you have to use a KafkaProducer to generate inputs for the source topics which feeds the KStream.

But, the intermediate topics can be populated, via

  • KafkaStreams#to
  • KafkaStreams#through
0
votes

You can use JXL(Java Excel API) to write a producer which writes to a kafka topic from an excel file. Then create a kafka streams application to consume that topic and produce to another topic. And you can use context.getTopic() to get topic from which the processor is receiving. Then set multiple if statements to call the process logic for that topic inside the process() function.