Kafka Consumer does not receive data when one of the brokers is down

Question

Kafka Quickstart
Using Kafka v2.1.0 on RHEL v6.9

Consumer fails to receive data when one of the Kafka brokers is down.

Steps performed:
1. Start zookeeper
2. Start Kafka-Server0 (localhost:9092, kafkalogs1)
3. Start Kafka-Server1 (localhost:9094, kafkalog2)
4. Create topic "test1", num of partitions = 1, replication factor = 2
5. Run producer for topic "test1"
6. Run consumer
7. Send messages from the producer
8. Receive messages on the consumer side.

All the above steps worked without any issues.

When I shutdown Kafka-Server0, the consumer stops getting data from Producer. When I bring back up Kafka-Server0, the consumer starts to get messages from where it left off.

These are the commands used

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test1
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test1

The behavior is the same (no message received on the consumer side) when I run the consumer with two servers specified in the --bootstrap-server option.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9094 --topic test1

Any idea why the consumer stops getting messages when server0 is down even though the replication factor for the topic test1 was set to 2?

There is a similar question already but it was not answered completely Kafka 0.10 quickstart: consumer fails when "primary" broker is brought down

Answer 1

If the offsets topic is unavailable, you cannot consume.

Look at the server.properties file for these, and see the comment above, and increase accordingly (only applies if topic doesn't already exist)

# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1

According to your previous question, looks like it only has one replica

See how you can increase replication factor for an existing topic

Answer 2

In initial versions of Kafka, offset was being managed at zookeeper, but Kafka has continuously evolved over the time introducing lot of new features. Now Kafka manages the offset in a topic __consumer_offsets.

You can think of a scenario where you created a topic with a replication factor of 1. In case the broker goes down the data is only on that Kafka node which is down. So you can't get this data. Same analogy applies to __consumer_offsets topic.

You need to revisit the server.properties in order to get features you are expecting. But in case you still wanna consume the messages from the replica partition, you may need to re-start the console consumer with --from-beginning true