How Kafka leader replica decides to advance Highwater Mark (HW) when replicating data to follower replicas

3
votes

I read about Kafka replication protocol. I found that Kafka maintains LEO and HW. As I understood,

LEO: Offset of latest message a replica has seen.

HW: Offset of the latest message which is guaranteed that each replica has seen.

Kafka producer can be set to these acknowledgement methods.

  1. acks = 0
  2. acks = 1
  3. acks = all

So my question is how the leader advances the HW depending on the acknowledgment method Kafka producer uses.

What I understood was,

  1. for acks = 0, Leader advances the HW when it sees a new message.

  2. for acks = 1, Leader advances the HW when it wrote new message to its local log.

  3. for acks = all, Leader advances HW when each an every follower sent ack that they got the message.

Is this correct? Can anyone clarify this?

1
apache-kafkareplicationdistributed-systemfault-tolerance

1 Answers

2
votes

With further exploring, I found that there is no relation between HW advancement and acknowledgment method set in broker configurations. Regardless of what is set to acks, the leader replica waits for all the other followers to persist the message before advancing the HW.

  • When acks = 0 and acks = 1, leader sends ack to the producer once it has seen a message and once it has persisted the message to its local log respectively. But waits for all the other replicas to persist the message before setting LEO to HW.
  • If acks = all and min.insync.replica = replication factor, the leader needs to wait for all replicas to persist the message before sending ack to the producer. Only In this case, leader advances the HW sychronous with the ack sent to producer.