A question related to topic compaction. In a compacted topic, when there is a delay with the log cleaner cleaning up the previous offsets (3,4,5) for a particular key (assuming 5 is the latest offset ), while consuming those offsets as a consumer will I see only the latest offset (5) for that key even when 3 and 4 are still not compacted yet or the consumer will get (3,4,5 ) in that order?
0
votes
1 Answers
1
votes
Your consumer will get (3,4,5) if 3 and 4 have not yet been compacted out of the topic. This is the same result you would get if you subscribed when 3 was the latest message and then 4 and 5 where subsequently published so it should not break your consumer logic. Also compaction never runs on the most recent segment of each topic partition because those segment files are still open for writes. This means that there is no guarantee that each key exists on only exactly one message.
