How can I run Kafka Consumer processor instance on multiple nodes with Apache Nifi

0
votes

Currently we are using Apache NiFi to consume messages via Kafka consumer. Output of kafka consumer is connected to hive processor.

I'm looking into how to run kafka consumer instance on a nifi cluster.

I have 3 nodes of nifi cluster and a kafka topic which have 3 partitions, I want the kafka consumer to be able run on each node so each consumer can poll message from one of topic partitions.

After I started the kafka consumer processor ,i can only see that the kafka consumer always run on a single node but not all nodes.

Is there any configuration that I missed?

1
apache-kafkaapache-nifi
How many kafka brokers do you have? have you specified all of them on 'ConsumeKafka' processor? - Pushkr
My Kafka cluster have more than 20 brokers, and I have configured all of them in the ConsumeKafka processor - jason zhang

1 Answers

0
votes

NiFi uses the Apache Kafka client which is what performs the assignment of consumers to partitions. When you start the processor, assuming you have it set to 1 concurrent task, then you should have 1 consumer on each node of your cluster, and each consumer should get assigned a different partition.

https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka