Apache Kafka consumer program in Java

Question

I am very new to Apache Kafka. I am given a task to write a consumer that will consume messages from topic(s). Below are my queries:

Should I tell my consumer Java program by providing the topic name explicitly (by passing as argument to main method) or is there any way by which it automatically discovers the topics?
I read somewhere that consumer program should be multi-threaded and after going through this link https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example, I think that it is creating number of threads for reading from single topic.

I thought what multi-threaded means is that my consumer program should spawn as many threads as topics are. If I have to read messages from, say, 3 topics, then do I need to start 3 consumer programs one for each topic? Can't I do the same in single program? Please clarify.
I read there are two types of consumer programs: Simple and High Level. I don't know which one to write.

How can I get started on this?

user2720864 user2720864 · Accepted Answer · 2014-07-21T08:23:51

1) Getting list of topic from the Kafka cluster could be a tricky job as there are no such direct api available for this .. the best bet is to query zookeeper as Kafka use it to maintain all of its states, this links explains about the zookeeper data structure. You can read this link for a similar discussion

2) the example assumes that you will use it to fetch data from a given topic, you might try to create a list of topics and loop through them to see how it goes, some thing like

     for(String topic : topicList) {
         new ConsumerGroupExample(zooKeeper, groupId, topic).run(threads);
     }

but not sure about the exact outcome of this, would be great if you could share your finding

3) Depends how much control you want over the topics based on your use cases. The high level does a lot of lot of bookkeeping and error handling but does not allow fine grain access over topic such as reading same message multiple times, consuming particular subset of messages etc where as all these functionality is provided by the simple consumer but in that case you need to handle cases like topic offset management, broker and leader detection etc.
You should take a detail look in their consmer documentation for a much in-dept understanding of the same.

Apache Kafka consumer program in Java

1 Answers