kafka multi node set up active-Active

0
votes

I am new to kafka and have some basic doubts. I was trying to do a multinode kafka set up (Active-Active). here is the scenario :-

  1. running single zookeeper (port: 2181)
  2. started three kafka brokers (9092,9093,9094)
  3. create a single topic test (partition: 1, replication 3)
  4. started producer using broker list (9092,9093)
  5. started consumer on 9092,9093,9094, but all of them are using same topic.

Now my doubts are, when I am sending a message from producer it is consumed by all three consumer,

q1. does that mean it is duplicating the message ?

q2. as producer is running on 9092,9093 then why consumer on port 9094 is consuming the message ? is it because it is using same topic name ?

q3. how I can make sure when i am sending a message from producer it should be consumed by a single consumer ? (I tried grouping but if any consumer is in different group it is receiving the message.)

q4. if I want to distribute the messages between consumers, how I can achieve this ?

q5. any other way to set up a multi node kafka (Active-Active) with single producer and multiple consumers ?

any help is appreciated.

1
javaapache-kafkakafka-consumer-api
1. Your consumers must be in different groups. If you don't want the messages to be read multiple times, set group.id – dawsaw
2. Ports are server ports. You are running 3 separate brokers that talk to each other. Which one you bootstrap to doesn't matter, you'll be routed to the one you need in both producing in consuming. – dawsaw
3. You must use consumer groups for this. – dawsaw
4. Not sure what this means. I think you are again referring to not duplicating between consumers, which means leveraging consumer groups. – dawsaw
5. It looks like you are in the right track, you just need to read more about consumer groups. – dawsaw

1 Answers

0
votes

I've answered in comments, but this is really a question coming down to a lack of understanding about consumer groups in Kafka. This is too long to discuss in this format, so I'll direct you here https://kafka.apache.org/documentation/#theconsumer