Is it possible to kill a consumer from the Kafka server?

1
votes

When I check for consumer lag, it shows that a particular consumer-id is running from a particular host consuming from a topic. But when I go to that host there is no such consumer running.

How do I kill this consumer-id, so that I can reset consumer offset for the group that its part of.

Kafka server version: 0.11.0.1

Kafka client version(stuck): 0.10.0.2

This consumer-id got stuck in the first place as it was not able to consume messages because of some messages having headers in Kafka.

I've already tried the following:

  1. Consuming from a different host and different Kafka version, it consumes messages but the consumer-id, host does not change.
  2. Restarting kafka broker which is the leader for that topic.
  3. Changing security groups to prevent the host from connecting to my broker.
1
apache-kafkakafka-consumer-api
reset the consumer's host? – aran
@AsierAranbarri thanku for looking into this. I would try this as the last resort, as it is owned by another team. I was just curious if there is a possible solution from my end as someone who owns the Kafka cluster, rather than asking clients to reboot their hosts. – Jobin Ar
well, you could set SASL as security, so if there's still some consumer hanging, it will be kicked out. Or maybe deleting consumer group, and after it, restarting the broker, works (maybe the last step is not needed). For example: stackoverflow.com/a/51925498/2148953 – aran
Just saw that your version <1.0, so you can't delete consumer groups like that : ( – aran
Yeah, cannot delete consumer groups. Not exactly sure why setting SASL as security will kick out the hanging consumer, but will definitely give it a try(if possible as this is a production cluster). – Jobin Ar

1 Answers

1
votes

Perhaps what you see is not a consumer id, but a consumer group, see Kafka docs, consumer config to learn about the difference.

Kafka uses consumer groups to keep track of the last consumed message (consumer offset), so when talking about the consumer lag this is probably the explanation.

This means there is no consumer running and you only need to get rid of the consumer offset for this group. See e.g. How do I delete a Kafka Consumer Group to reset offsets?