Confluent 5.0.0 kafka connect hdfs sink: Cannot describe the kafka connect consumer group lag after upgrade

0
votes

We upgraded from Confluent 4.0.0 to 5.0.0, after upgrading we cannot list the kafka connect hdfs sink connector consumer lag.

$ /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <hostname>:9092 --list | grep scribe_log_backend
Note: This will not show information about old Zookeeper-based consumers.
connect-kconn_scribe_log_backend
$ /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <hostname>:9092 --group connect-kconn_scribe_log_backend --describe
Note: This will not show information about old Zookeeper-based consumers.
$

Were there any modifications done to the consumer group command in kafka 2.0/confluent 5.0.0 ? How do i track the lag we need to alert based on this lag? Our brokers run on kafka version 1.1.0. Also cannot see the connect consumer group in kafka manager after the upgrade. There is no issue with kafka connect as the connector is able to write to hdfs. Thanks.

1
apache-kafkaconfluent-platform

1 Answers

0
votes

Kafka Connect HDFS connector no longer commits offsets so there is nothing to base lag calculation on.

PS. The recovery is based on file naming convention is HDFS, the file names have partition and offset info.