Delete __Consumer_offset Topic form Kafka

Question

I'm trying to delete Kafka topic __Consumer_offset as it is causing a lot of confusion for my brokers. When i do so, it says this topic can't be marked for deletion.

i'm using the zookeeper cli to delete it such as rmr /brokers/topic __consumer_offset, but it is not working!

Answer 1

__consumer_offsets is a kafka internal topic and it is not allowed to be deleted through delete topic command. It contains information about committed offsets for each topic:partition for each group of consumers (groupID). If you want to wipe it out entirely you have to delete the zookeeper dataDir location. That implies, you lose all the metadata.

Also if you just want to get rid of the existing consumer groups, you can as well reset the offsets or consider deleting them.

Answer 2

AFAIK you cannot delete that topic. It is a internal topic and should not be deleted manually.

If it is must, then you will have to manually clean/remove your data directory. When you deploy Kafka brokers and Zookeepers it creates data directory.

Note: By removing data directory you will loose all topics and related data. So this is not feasible option in Production.