how producers find kafka reader

5
votes

The producers send messages by setting up a list of Kafka Broker as follows.

props.put("bootstrap.servers", "127.0.0.1:9092,127.0.0.1:9092,127.0.0.1:9092");

I wonder "producers" how to know that which of the three brokers knew which one had a partition leader. For a typical distributed server, either you have a load bearing server or have a virtual IP, but for Kafka, how is it loaded? Does the producers program try to connect to one broker at random and look for a broker with a partition leader?

2
apache-kafka

2 Answers

4
votes

A Kafka cluster contains multiple broker instances. At any given time, exactly one broker is the leader while the remaining are the in-sync-replicas (ISR) which contain the replicated data. When the leader broker is taken down unexpectedly, one of the ISR becomes the leader.

Kafka chooses one broker’s partition’s replicas as leader using ZooKeeper. When a producer publishes a message to a partition in a topic, it is forwarded to its leader.

According to Kafka documentation:

The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of the partitions. Each partition is replicated across a configurable number of servers for fault tolerance.

Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.

You can find topic and partition leader using this piece of code.

EDIT:

The producer sends a meta request with a list of topics to one of the brokers you supplied when configuring the producer.

The response from the broker contains a list of partitions in those topics and the leader for each partition. The producer caches this information and therefore, it knows where to redirect the messages.

1
votes

It's quite an old question but I have the same question and after researched, I want to share the answer cuz I hope it can help others.

  • To determine leader of a partition, producer uses a request type called a metadata request, which includes a list of topics the producer is interested in.
  • The broker will response specifies which partitions exist in the topics, the replicas for each partition, and which replica is the leader.
  • Metadata requests can be sent to any broker because all brokers have a metadata cache that contains this information.