How exactly Apache Nifi ConsumeKafka_1_0 processor works

Question

I have Nifi cluster of and Kafka is also installed there.

Created one topic with 5 partitions, start consuming that topic with one gourp-id. So that each partition will get unique messages.

Now I created the 5 ConsumeKafka_1_0 processors having the intent of getting unique messages on each consumer side. But only 2 of the ConsumeKafka_1_0 are consuming all the messages rest is setting ideal.

Now what I did is started the 5 command line Kafka consumer, and what happened is, I was able to see the all the partitions are getting the messages and able to consume them from command line consumer in round-robin fashion only.

Also, I tried descried the Kafka group and what I saw was only 2 of the Nifi ConsumeKafka_1_0 is consuming all the 5 partitions and rest is ideal, see the snapshot.

Would you please let me what I am doing wrong here with Nifi consumer processor.

Note - i used Nifi version is 1.5 and Kafka version is 1.0.

Answer 1

I've written this article which explains how the integration with Kafka works:

https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka

The Apache Kafka client (used by NiFi) is what assigns partitions to the consumers.

Typically if you had a 5 node NiFi cluster, with 1 ConsumeKafka processor on the canvas with 1 concurrent task, then each node would be consuming 1 partition.