Current offset behavior when set by kafka-consumer-groups to earliest?

4
votes

I have a kafka topic with 25 partitions and the cluster has been running for 5 months.

As per my understanding for each partition for a given topic, the offset starts from 0,1,2... (un-bounded)

I see log-end-offset at a very high value (right now -> 1230628032)

I created a new consumer group with offset being set to earliest; so i expected the offset from which a client for that consumer group will start from offset 0.

The command which I used to create a new consumer group with offset to earliest:

kafka-consumer-groups  --bootstrap-server <IP_address>:9092 --reset-offsets --to-earliest --topic some-topic --group to-earliest-cons --execute

I see the consumer group being created. I expected the current-offset being to 0; however when I described the consumer group the current offset was very high , at the moment --> 1143755193.

The record retention period set is for 7 days (standard value).

My question is why didn't we see the first offset from which a consumer from this consumer group will read 0? Has it to do something with data-retention?

Can anyone help understand this?

1
apache-kafka

1 Answers

6
votes

It is exactly data retention. It is highly probable that Kafka already removed old messages with offset 0 from your partitions, so it doesn't make sense to start from 0. Instead, Kafka will set offset to the earliest available message on your partition. You can check those offsets using:

./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list <IP_address>:9092 --topic some-topic --time -2

You will probably see values really close to what you're seeing as new consumer offset.

You can also try and set offset explicitly to 0:

./kafka-consumer-groups.sh  --bootstrap-server <IP_address>:9092 --reset-offsets --to-offset 0 --topic some-topic --group to-earliest-cons --execute

However, you will see warning that offset 0 does not exist and it will use higher value (aforementioned earliest message available)

New offset (0) is lower than earliest offset for topic partition some-topic. Value will be set to 1143755193