kafka-consumer-groups CLI not showing node-kafka consumer groupf

Question

I have a kafka consumer group running on node.js powered by node-kafka. When this consumer group is active or in-active, I expect to see it reported by the kafa-consumer-groups CLI.

The kafka-consumer-groups CLI does show the console consumers and not just the node consumer.

I can see the node consumer group in Kafka Tool. It doesn't show up in the Kafa-consumer-groups CLI output

kafka-consumer-groups --bootstrap-server localhost:9092 --list kafka-consumer-groups --bootstrap-server localhost:9092 --group node-kafka-consumer --describe

kafka-consumer-groups CLI should show all consumers - console and programmatic (in my case node-kafka consumer)

Have you tried using --zookeeper instead of bootstrap-server? — OneCricketeer
I did but didn't work (At least with my Kafka - 2.3.1). With the above code offsets are being written to zookeeper and that should explain why kafka-consumer-groups do not list the consumer group. I found an alternate way with kafka-node that would write offsets to kafka. I would post below. — cubsnlinux

cubsnlinux cubsnlinux · Accepted Answer · 2019-11-11T04:47:40

Here is the solution that uses kafka-node ConsumerGroup object to write offsets to kafka instead of zookeeper

const { ConsumerGroup } = kafka;


const consumerOptions = {
  kafkaHost: 'localhost:9092',
  groupId: 'kafka-node-consumer-group',
  protocol: ['roundrobin'],
  fromOffset: 'earliest'
};

const topics = ['zoo_animals'];

const consumerGroup = new ConsumerGroup(
  { id: 'node-app-1', ...consumerOptions },
  topics
);

consumerGroup.on('message', onMessage);
consumerGroup.on('error', onError);

function onMessage(message) {
  console.log('message', message);
}

function onError(error) {
  console.log('error', error);
}

process.once('SIGINT', function() {
  consumerGroup.close(true, err => {
    if (err) {
      console.log('error closing consumer', err);
    } else {
      console.log('closed consumer');
    }
  });
});```

kafka-consumer-groups CLI not showing node-kafka consumer groupf

1 Answers