Kafka Consumer subscribes to topic but doesn't consume messages and doesn't appear in consumer group list

0
votes

A Kafka Consumer (org.apache.kafka:kafka-clients:2.3.0) should consume messages from a topic with one partition.

The service logs the consumer subscription but it doesn't consume messages and doesn't appear in consumer group list using the Official Apache Tools kafka-consumer-groups.sh.

2020-06-15 11:50:50.308 [main] INFO  o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-1, groupId=my-groupid] Subscribed to topic(s): my-topic

Service configuration should be OK because it works in pre-production environment. The difference between the environments is that production has three hosts and pre-production just one and I'm sure that there aren't networking issues.

What might be the cause behind this?

1
javaapache-kafkaapache-zookeeperkafka-consumer-apipublish-subscribe
Any good reason why you're not sharing your code? – Giorgos Myrianthous
@GiorgosMyrianthous think it's unnecessary because I'm sure the code works. I'm looking for some higher level suggestions. What part of code should I share to enrich my question? – batodev
Is topic "healthy", i.e. it has ISR > min.insync.replica? Any diff in acks on producer side? – mazaneicha
This log message is printed when you call subscribe(). It does not indicate the consumer is part of the group, is consuming or is even connected as you can call subscribe() without calling poll()! Please attach some code and/or the full client logs – Mickael Maison

1 Answers

0
votes

The problem was zookeeper configuration. It was configured in 3 independent nodes handling 3 Kafka brokers and not in a distributed system.