2
votes

I am trying to replicate Kafka cluster with MirrorMaker 2.0. I am using following mm2.properties:

# mm2.properties
clusters=source, dest

source.bootstrap.servers=kafka-1:9092,kafka-2:19092,kafka-3:29092
dest.bootstrap.servers=kafka-4:39092,kafka-5:49092,kafka-6:59092

source->dest.enabled=true

I can see that when MirrorMaker starts, it creates its own topics. For example on source cluster it creates: mm2-configs.dest.internal with following content:

{
   "properties":{
      "connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
      "source.cluster.producer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
      "source.cluster.alias":"dest",
      "source.cluster.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
      "target.cluster.producer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
      "enabled":"false",
      "target.cluster.admin.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
      "target.cluster.alias":"source",
      "target.cluster.consumer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
      "name":"MirrorSourceConnector",
      "target.cluster.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
      "source.cluster.admin.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
      "source.cluster.consumer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092"
   }
}

and at destination cluster mm2-configs.source.internal topic with following content:

{
   "properties":{
      "connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
      "source.cluster.producer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
      "source.cluster.alias":"source",
      "source.cluster.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
      "target.cluster.producer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
      "enabled":"true",
      "target.cluster.admin.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
      "target.cluster.alias":"dest",
      "target.cluster.consumer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
      "name":"MirrorSourceConnector",
      "target.cluster.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
      "source.cluster.admin.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
      "source.cluster.consumer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092"
   }
}

But when I am trying to send messages to source cluster (kafka-1,kafka-2,kafka-3) - nothing copied to the second cluster.

What do I miss?

Appreciate any help Thx,

2
How about starting simpler? Remove Docker. Run one broker per physical host?OneCricketeer
@cricket_007 Since I have only my laptop, do you mean on VMs? And, by the way, the old MirrorMaker (1.x) works OK with dockersMikhail Grinfeld
Anything can work with Docker, sure. My point is that you should create a minimal reproducible exampleOneCricketeer
@cricket_007 I understand what do you mean :) But I have only one physical computer (my laptop) and I need at least 2, and I thought that Docker is the solution for this. But I'll try with only one broker, thxMikhail Grinfeld

2 Answers

0
votes

Your config files look ok. However, those bootstrap server ports look off. 19092, 29092, etc. Those are typically 9092 only. Can you try changing those back to 9092?

0
votes

I think, I found (with help of my friend) what was the problem. In case when topic does NOT exist before starting MirrorMaker, replication won't happen. Topic should exist in source Kafka cluster.

I used Java producer with topic auto create enabled. Seems, automatic topic creation, doesn't trigger MirrorMaker to start replication.

I don't really know if it's as designed behavior when defining source->dest.enabled=true or maybe related to solved few days ago bug: https://github.com/apache/kafka/pull/7745.