Why is my kafka topic not consumable with a broker down?

Question

My issue is that I have a three broker Kafka Cluster and an availability requirement to have access to consume and produce to a topic when one or two of my three brokers is down.

I also have a reliability requirement to have a replication factor of 3. These seem to be conflicting requirements to me. Here is how my problem manifests:

1. I create a new topic with replication factor 3
2. I send several messages to that topic
3. I kill one of my brokers to simulate a broker issue
4. I attempt to consume the topic I created
5. My consumer hangs
6. I review my logs and see the error: Number of alive brokers '2' does not meet the required replication factor '3' for the offsets topic

If I set all my broker's offsets.topic.replication.factor setting to 1, then I'm able to produce and consume my topics, even if I set the topic level replication factor to 3.

Is this an okay configuration? Or can you see any pitfalls in setting things up this way?

Answer 1

You only need as many brokers as your replication factor when creating the topic.

I'm guessing in your case, you start with a fresh cluster and no consumers have connected yet. In this case, the __consumer_offsets internal topic does not exist as it is only created when it's first needed. So first connect a consumer for a moment and then kill one of the brokers.

Apart from that, in order to consume you only need 1 broker up, the leader for the partition.