Apache Kafka Cluster Consumer Fail Over

Question

I had a set up of 3 Node Zookeeper and 3 Broker Cluster when one of my brokers goes down in Cluster, the producer is not giving any Error but, consumers will throw an error saying that...

Marking coordinator Dead for the group... Discovered coordinator for the group.

According to my knowledge if any one Broker available across the cluster I should not be stopped consuming messages.

But, as of now Server.1, server.2, server.3 if my server.2 goes down my all consumers stops consuming messages.

What are the exact parameters to set to achieve failover of producers and as well as consumers?

You need to have topic replication of more than one (e.g. in your case, maybe 2), so that even if a broker goes down, the replica topic will be available on some other broker, assuming of-course both of them don't go down at the same time. It depends of how much of high availability you can ensure. — Bitswazsky

OneCricketeer OneCricketeer · Accepted Answer · 2018-11-27T18:34:31

if my server.2 goes down my all consumers stops consuming messages.

For starters, you disable unclear leader election in the brokers, and create your topics with --replication-factor=3 and a configuration of min.insync.replicas=2.

To ensure that a producer has at least two durable writes (as set by the in-sync replcicas), then set acks=all

Then, if any broker fails, and assuming a leader election does not have any error, a producer and consumer should seemlessly re-connect to the new leader TopicPartitions.

Apache Kafka Cluster Consumer Fail Over

1 Answers