Kafka message loss and Acks=all

0
votes

I understand if Acks=all and min ISR is set to >= 2 reduces the chance of message loss as more than one broker has the message in memory. What happens when there is a simultaneous failure of connectivity between producer and brokers 1 before the message is acked but request sent to leader for write. 2 message were written but failed during ack

Is there any chance of message loss ?

1
apache-kafkakafka-producer-api
'failure of connectivity between client and brokers'. Did you mean producer when you say 'client' ? - ADS
Edited to reflect producer instead of client - Balan
A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write - Deadpool
@Deadpool I have mentioned those assumptions in the question already. I am curious about connectivity loss after. Write request is sent and or finished but producer has not processed the response. Is it only loss of ack ? - Balan
Actually producer stores these messages into buffer and buffer is responsible of sending these messages as batch to kafka cluster, and to your point is it only loss of ack yes but that might be very rare @Balan - Deadpool

1 Answers

2
votes

When producer send data to brokers, actually it send only to only one - the leader. Leader broker should confirm that message has been received. If you configure broker as described, the leader will await acknowledgments from replicas. If something goes wrong, the leader don't send its own confirmation. When producer doesn't receive acknowledgment from the leader, it will understand that message has NOT been delivered. For simplicity, I dropped out that there are some retires before any failure message.

Assuming that,

  1. Since producer doesn't receive ack, it will re-send original message. If connection has been restored then broker could see that this message has been received already. Then broker send ack to producer.

  2. If message was written to the leader but replicas was failed to ack, then leader will treat such message as not saved and remove from its storage.