I have a producer that publishes messages on a topic called 'mytopic' just fine. I have 2 consumers in 2 different consumer groups listening for these messages. I started these 2 consumers and producer in following sequence.
1) Start consumer 1 in group 'group1' 2) start producer to publish several hundreds messages
After sometime I check the offset of consumer 1, which is as I expect:
/opt/kafka_2.11-0.10.0.0/bin/kafka-consumer-offset-checker.sh --zookeeper localhost:2181 --topic mytopic --group group1
Output:
Group Topic Pid Offset logSize Lag Owner
group1 mytopic 0 30230 36942 6712 none
3) Now I start consumer 2 in group 'group2' to listen to the same messages but it comes back with 0 messages on every poll() call. The offset check for this consumer shows me that its offset is same as the logSize.
/opt/kafka_2.11-0.10.0.0/bin/kafka-consumer-offset-checker.sh --zookeeper localhost:2181 --topic mytopic --group group2
Output:
Group Topic Pid Offset logSize Lag Owner
group2 mytopic 0 36942 36942 0 none
Same problem for any other consumer with a new consumer group. Why is the consumer joining a new consumer group after messages have been published not seeing the old messages even though messages exists on the topic (ie., haven't been deleted)?
