Kafka Mirror Maker not Replicating topics when source and destination clusters are not in the same configuration

1
votes

We have setup Mirror Maker to replicate topics between two kafka clusters when the source and destination setup are different(source= 2 servers , destination = 3 servers). when starting the MirrorMaker its throws the following error: "Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)" and no replication between the clusters. we used MirrorMaker version 2.

Can anyone please provide some suggestion why it is not working as expected?

1
apache-kafkaapache-kafka-connectkafka-producer-apiapache-kafka-mirrormaker
Does it work if you use the same cluster size, e.g. 2 servers for both source and destination? – Amir Kirsh
Please show your configuration files. The error is suggesting that that min ISR for the destination cluster isn't being met – OneCricketeer

1 Answers

1
votes

Mirrormaker 2.0 documentation is only coming in the new 2.5.0 release - which makes setting up Mirrormaker 2.0 frustratingly hard. Basically I think you would need to set your replication factors (which defaults to 3 in Kafka) to 2 as this is your minimum for one cluster. These are the config values you probably need to set:

  • checkpoints.replication.factor
  • heartbeats.replication.factor
  • offset-syncs.replication.factor
  • config.storage.replication.factor
  • offset.storage.replication.factor
  • status.storage.replication.factor
  • replication.factor
  • offsets.replication.factor
  • transaction.state.log.replication.factor

You can set these in the MM2 config file.