Does Kafka producer write keyless message to another partition when a partition is down?

2
votes

For example, I have a topic that has 2 partitions and a producer using defaultpartitioner (round-robin I assumed) writes to the topic. At some point, partition 1 becomes unavailable because all of the replica brokers go offline. Assuming the messages have no specified keys, will the producer resend the messages to partition 2? or simply gets stuck?

1
apache-kafkakafka-producer-api

1 Answers

0
votes

That is an interesting question and we should look at it from a broader (cluster) perspective.

At some point, partition 1 becomes unavailable because all of the replica brokers go offline.

I see the following scenarios:

  1. All replica brokers of partition one are different to the replica brokers of partition two.
  2. All replica brokers of partition one are the same as for partition two.
  3. Some replica brokers are the same as for partition two.

In scenario "1" it means you still have enough brokers alive as the replication factor is a topic-wide not a partition-based configuration. In that case as soon as the first broker goes down its data will be moved to another broker to ensure that your partition always has enough in-sync replicas.

In scenarios "2" both partitions become unavailable and your KafkaProducer will eventually time out. Now, it depends if you have other brokers that are alive and can take on the data of the partitions.

In scenario "3" the dead replicas would be shifted to running brokers. During that time the KafkaProducer will only write to partition 2 as this is the only available partition in the topic. As soon as partition 1 has enough in-sync replicas the producer will start producing again to both partitions.

Actually, I could think of many more scenarios. If you need a more concrete answer you need to specify

  • how many brokers you have,
  • what your replication factor actually is and
  • in what timely order which broker goes down.

Assuming the messages have no specified keys, will the producer resend the messages to partition 2?

The KafkaProducer will not re-send the data that was previously send to partition 1 to partition 2. Whatever was written to partition 1 will stay in partition 1.