Confluent Kafka Replicator | Consumer group offsets not replicated to destination cluster

1
votes

Update:

Found a workaround by setting fromBeginning: false for the consumer on the destination cluster. Not a best solution but it works...

For some reason all consumer group offsets not replicated to destination cluster. For example I started a consumer on source cluster with consumer group TEST (all messages are pulled) then I started a consumer (with consumer group TEST) on the same topic that was replicated into destination cluster, the result is that both consumers got all message.

I read about consumer-offset-translation-feature but didn't made it work.

Consumer Config:

bootstrap.servers=sourceKafkaEndpoint:9092
interceptor.classes=io.confluent.connect.replicator.offsets.ConsumerTimestampsInterceptor

Producer Config:

bootstrap.servers=destKafkaEndpoint:9092

Replicator Config:

connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
auto.offset.reset=latest
topic.config.sync=false
config.action.reload=none

Executable Command:

replicator --consumer.config consumer.properties --producer.config producer.properties --replication.config replication.properties --topic.regex ".*" --cluster.id replicator

Kafka Info:

Source cluster Kafka version: 2.0.1-cp4
Destination cluster Kafka version: 2.4.1

any ideas?

1
apache-kafkaconfluent-platform

1 Answers

2
votes

It's an old question and a lot of things have been change since I guess, but it's for the better because now Replicator is documented a lot better.

As explained here:

Replicator will not copy the internal __consumer_offsets or __transaction_state topics from the source cluster even if matched in topic.regex. To copy these topics, list them in topic.whitelist.

This is because consumer group offsets are meaningless between clusters. You can not rely on them being identical. See my other post for more detail.
The only reliable way to achieve such a task is using offset translation via timestamp preservation. It allows to translate __consumer_offsets to the destination cluster based on the message creation time from the source cluster.