I would like to set offset of a topic mytopic with 1 partition and given group ID testgroup1 to 0. However it is not always possible. If I want to set offset to 0 I get this message:
bash-4.4# kafka-consumer-groups.sh --bootstrap-server localhost:9092 --topic mytopic --group testgroup1 --reset-offsets --to-offset 0 --execute
[2021-06-04 09:23:30,854] WARN New offset (0) is lower than earliest offset for topic partition mytopic-0. Value will be set to 1365671 (kafka.admin.ConsumerGroupCommand$)
bash-4.4# kafka-topics.sh --bootstrap-server localhost:9092 --topic mytopic --describe
Topic: mytopic PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: mytopic Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
bash-4.4# kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-name mytopic --entity-type topics
Dynamic configs for topic mytopic are:
bash-4.4#
In Kafka log I can see this after whole topic was consumed; not sure if it's really related:
[2021-06-04 10:18:36,130] INFO [Log partition=__consumer_offsets-19, dir=/kafka/logs] Deleting segment files LogSegment(baseOffset=0, size=634, lastModifiedTime=1598954190000, largestRecordTimestamp=Some(1585909899136)) (kafka.log.Log)
[2021-06-04 10:18:36,131] INFO Deleted log /kafka/logs/__consumer_offsets-19/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2021-06-04 10:18:36,132] INFO Deleted offset index /kafka/logs/__consumer_offsets-19/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2021-06-04 10:18:36,132] INFO Deleted time index /kafka/logs/__consumer_offsets-19/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
It is not even possible to consume topic again using this command:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --group testgroup1 --topic mytopic
I've read other questions like:
- Can't reset kafka topic in a group to 1
- Current offset behavior when set by kafka-consumer-groups to earliest?
- How to change start offset for topic?
But I haven't found a reason why Kafka behaves like that, i.e. earliest offset to set to different value and it is not possible to return to offset 0 again. Maybe it has something to do with data retention but I've tried to set log retention to 3 years:
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 86400000
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.max.compaction.lag.ms = 9223372036854775807
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = /tmp/kafka-logs
log.dirs = /kafka/logs
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 60000
log.flush.scheduler.interval.ms = 9223372036854775807
log.flush.start.offset.checkpoint.interval.ms = 60000
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.downconversion.enable = true
log.message.format.version = 2.7-IV2
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 300000
log.retention.hours = 26280
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 6000
