Mirror Maker not making topic automantically

Question

I am setting a mirror maker between 2 clusters of kafka .

Kafka Version: kafka_2.11-2.1.0

My server.properties has auto.create.topics.enable=true .

I am running mirror maker on my target cluster .

But the new topics don't get created automatically when source cluster adds a topic . I have to end up creating them manually . And even after creating manually , the records from source cluster don't get populated in this new topic , until or unless i don't restart my mirror maker . Once restarted , it starts replicating immediately post insert of data on source .

Mirror Maker command

./kafka-mirror-maker.sh --consumer.config consumer.properties  --producer.config producer.properties  --whitelist '.*'  --num.streams 2

Consumer.properties

bootstrap.servers=!source_host_name_with_port!
group.id=consumer_replicator_group
client.id=mirror_maker_consumer
exclude.internal.topics=true

Producer.properties

bootstrap.servers=!destination_host_name_with_port!
acks=1
batch.size=100
client.id=mirror_maker_producer

Any ideas please.

OneCricketeer OneCricketeer · Accepted Answer · 2019-03-04T03:18:26

It's documented on the Kafka website to create the topics on the destination first. If you don't, then you get the default partition and replication counts on the destination cluster (which often do not match the source topic)

Mirror Maker not making topic automantically

1 Answers