Why Nifi consumerKafka_0_10 Processor receives flowfile less than total flowfile?

Question

I have 1 producers (PublishKafka_0_10 processor) and 1 consumer (ConsumerKafka_0_10 processor) to receive flowfile from Kafka cluster.

I see on Nifi UI admin, the total out of producers is 7 packages but the consumer just receives only 4 packages. I also use kafka_console_consumer.sh to view the packages from producer and it displays whole 7 packages.

I don't know why and where I lost 3 packages from consumerKafka_0_10 processor.

I use kafka cluster with 3 nodes and nifi cluster with 3 nodes too.

Answer 1

A couple of things to check...

The ConsumeKafka processor defaults to latest offset the first time you run it, so if you had start PublishKafka first, and then ConsumeKafka, its possible that a few messages got published before the consumer started, and then the consumer is start at the offset of message 4.

Also check if you have a Message Demarcator in ConsumeKafka. If you do then it will be placing more than one message into a flow file.