kafka consumer group getting duplicate message in kafka-node

Question

I am trying to setup the Kafka queue for my application in node. I have 3 consumer in a consumer group and which is subscribed to single topic but when a producer sends messages to a topic having 3 partition, the consumer group receiving duplicate messages e.g consumer1 reads from partition1 and fetches messages and consumer2 also reads from same partition and fetches same message.

i have given the link to git repository which has all the executable code. i am not able to find is it setup problem? or node package problem.

options provided to create a consumer group is below

    consumerOptions = {
          'kafkahost:'localhost:9092',
          groupId: groupName,
          autoCommit: true,
          autoCommitIntervalMs: 1000,
          sessionTimeout: 15000,
          fetchMaxBytes: 10 * 1024 * 1024, // 10 MB
          protocol: ['roundrobin'],
          fromOffset: 'earliest'
    };

    var consumer = new kafka.ConsumerGroup(Object.assign({ id: 'consumer1' }, consumerOptions), topicName);

plz refer this link for executable code, broker config and zookeeper config https://github.com/Raghavendralacharya/kafka-node.git

Alexadreison Alexadreison · Accepted Answer · 2019-09-03T11:40:58

0

votes

Try out to change the fromOffset config to:

fromOffset: 'latest'

kafka consumer group getting duplicate message in kafka-node

1 Answers