Mirror Maker 2 topic mirroring scenarios for Producers

1
votes

I have a question about the failover in the scenario. Let's say that we have two clusters K1 and K2 and configured MM2 replication for TOPIC1 (originally created in just K1).

We configured the active-active replication:

K1->K2.enabled = true
K2->K1.enabled = true
K1->K2.topics = .*
K2->K1.topics = .*

On starting mirror maker 2, I see that topics are replicated from cluster K1 to K2 in the naming format K1.topic_name_here and vice versa for topics from cluster K2 to K1.

I see that there was no topic TOPIC1 created in K2, only K1.TOPIC1 was created. I see this scenario working for consumers, as in the beginning the consumers consume TOPIC1 from cluster K1. When the cluster K1 out of service, fail over happens. Consumers start to consume K1.TOPIC1 from K2.

My questions are as follows:

  1. For producers, they won't producer to the topic K1.TOPIC1 in cluster K2, my question is how do the producers go about producing data. Do I need to manually create a topic TOPIC1 in cluster K2 and this topic will be used for producing messages to when failover happens.
  2. If the above scenario is true, how do I move back to my primary cluster K1. As, now the topic TOPIC1 in cluster K2 has digressed from the topic TOPIC in K1. How do we sync the messages in this scenario ?
1
apache-kafkaapache-kafka-mirrormaker

1 Answers

2
votes

One way to address that is to introduce the concept of "local" and "remote" topics.

In a cluster, a "local" topic is a topic that you created in this cluster. A "remote" topic is a topic that was created by MirrorMaker2 and is typically prefixed (or suffixed) with a cluster alias, for example K1.TOPIC1.

Then to prevent "local" and "remote" topics from diverging, have your producers always produce to "local" topics and have consumers subscribed to both "local" and "remote" topics.

In your example:

  • Producers always use TOPIC1 whether they are connected to K1 or K2.
  • Consumers subscribe to TOPIC1, K1.TOPIC1, K2.TOPIC1 (you can also use a pattern, such as .*TOPIC1)

If you want to preserve strict ordering, when switching clusters, you can make your consumers drain the remote topics first and then consume from the local topics.