Kafka Mirror Maker not Replicating Counsumergroup Offset

Question

We have setup MirrorMaker to replicate messages across two Kafka cluster. We have also set exclude.internal.topic=false in mirror maker consumer properties to replicate internal topic. I assume this will also replicate __consumer_offset topic which will in turn sync the consumer group offset in secondary cluster.

But when we start the consumer group in secondary cluster its starting consuming messages from the beginning, so looks like the consumer group offsets are not getting replicated in secondary cluster.

Can anyone please provide some suggestion as how can we sync the consumergroup offset in secondary cluster using MirrorMaker or any other solution ?

Answer 1

While at first sight replicating the __consumer_offsets topic may seem like a good idea it's actually not going to work as you would expect.

When replicating topics across different cluster, in most cases record's offsets will be different between the source and target clusters. Reasons why offsets diverge between 2 clusters include:

• duplicate messages from Mirror Maker due to its At Least Once semantics
• different message order due to retries
• first offset not zero on source cluster due to retention limits

This renders the content of __consumer_offsets useless in the target cluster.

Mirror Maker 2, which will be part of Kafka 2.4 (expected to release November/December 2019), provides a way to "translate" offsets so consumers can be migrated easily between clusters.