How stream processor (low level API) source processor gets data from topic?

Question

I am novice to kafka stream processor, and came across the key-concept of "topology".

I created source processor which reads from a "source-topic" like this:

Topology topology = new Topology();
topology.addSource("SOURCE", "source-topic");

The above snippet would create (if my understanding is correct) a source stream processor named "SOURCE" and would listen for kafka topic "source-topic".

I didn't write any code for this "SOURCE" stream-processor, how is it able to get the messages from kafka topic? Is it a "special" type of stream-processor which is taken care by kafka stream API itself?

Can anyone help me understand this?

Answer 1

A topology starts from a source node to get data from Kafka, then contains a bunch of processor nodes to perform transformations and finally ends with a sink node to write transformed data into Kafka.

addSource() will create a source node in your topology. A source node consumes records from the topic specified and passes them on to the next nodes in the topology. It does not do any other logic. Under the cover, a source node will start a Kafka Consumer to get the records.

The Kafka Streams API enables you to focus on your logic (in the processors) instead of dealing with Consumers and Producers.

Answer 2

topology.addSource("SOURCE", "source-topic");

Above "SOURCE" is just a name. You can refer the Topology class here for more details.

How it works internally is - When a topology is created, you need to define a source, sink and processor using below methods:

addSource() , addSink() and addProcessor()

These methods add the processor node in the topology and internally calls the InternalTopologyBuilder class to build the Topology Graph.

On streams.start() method, it invokes the processing of the topology.