Clear a Kafka consumer group with stuck members

Question

I have the problem that a lot of my Kafka clients from one consumer group did not shutdown correctly and thus the Kafka cluster thinks they are still connected. Thus, I cannot connect to the consumer group with the new version of my client. It will be stuck in the rebalancing step.

According to documentation they should be removed after session.timeout.ms or maximum group.max.session.timeout.ms. At first I tried to set session.timeout.ms to 30000 milliseconds (30 seconds), but it was not listed on the startup of Kafka. group.max.session.timeout.ms was set to 300000 milliseconds (5 minutes) at that point in time. The consumers were not deleted after 30 seconds.

After that I tried to reduce group.max.session.timeout.ms to 30000 milliseconds (30 seconds) and restarted Kafka. However, again all clients just remained in the consumer group.

Now, it's about 2 hours later and the clients are still attached to the consumer group.

I tried to delete the consumer group with:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --delete --group GroupName

which gives me:

* Group 'GroupName' could not be deleted due to: NON_EMPTY_GROUP

Unfortunately, there does not seem to be a --force flag.

Next, I tried to list all members of the consumer group with this command:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --members --group GroupName --describe

This gives me 40-50 consumer group members (all of them must be inactive, because the only active consumer is stopped).

Is there a way how I can get Kafka to purge all consumers from this group or force it to delete the whole consumer group?

The group list command should show the IP's of the attached clients. You'll need to find those and actively stop them before deleting a group. Otherwise, you might want to try restarting Zookeeper as well — OneCricketeer
Thanks, I think restarting Zookeeper was a big help (even though I am not sure why, because it's a not a zookeeper-based consumer group). When I restarted Zookeeper, it went down to 5 consumers. Then I restarted the whole server, which brought it to one consumer. Stopped all other Kafka consumers and voila, finally the group was empty. If you have an explanation, why Zookeeper also needs to be restarted, feel free to add it to an answer to this question. Actually, thinking about it now, I guess that the consumer processes were still active on the server... I never checked ps -A. — aufziehvogel
@Aufziehvogel - Is there any reference for 1 st point in your question i.e. CG did not shutdown correctly and therefore kafka cluster thinks they are still connected. — swapnil7

Teo Wang Wei Teo Wang Wei · Accepted Answer · 2019-01-16T11:24:34

With

Group 'GroupName' could not be deleted due to: NON_EMPTY_GROUP

meaning the consumer group is still actively connecting to the server You have to first kill the process before you delete the consumer group. This works for me.

ps -ef | grep GroupName | grep -v grep | awk '{print $2}' | xargs kill
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--delete --group GroupName

Clear a Kafka consumer group with stuck members

2 Answers