kafka partition and producer relationship

0
votes

I have a kafka cluster with three brokers and one topic with replication factor of three and three partitions. I can see that every broker has a copy of log for all partitions with the same size. There are two producers for this topic.

One day I reduced writing volume of one producer by half. Then I found that all three brokers' inbound traffic reduced which is expected, but only partition 1's leader node's out traffic reduced which I don't understand.

The partition leader's outbound traffic reduced because of replication. But each broker is the leader of one partition, why only one leader's outbound traffic reduced? Is it possible that the producer only writes content to one partition? while I don't think so.

Please help me explain it. The cluster is working fine now, but I need to understand it in case of potential problem. Inbound Traffic Outbound Traffic

1
hadoopapache-kafkakafka-producer-api
It depends. Could you check the offset for each partition to see if you are producing messages evenly? – amethystic
Thanks for your reminder. I checked the logsize of all three partition. Actually, they are not evenly produced. This is very interesting. The unbalancement is huge like 1:6. Maybe it happens because of what ravthiru mentioned, removing specific keys results in no data for particular partitions. – billtian

1 Answers

1
votes

Assuming you are using Default Partitioner for KafkaProducer, which means two events with the same key are guaranteed to be sent to the same partition.

From From Kafka Documentation

All reads and writes go to the leader of the partition and Followers consume messages from the leader just as a normal Kafka consumer would and apply them to their own log.

You could have reduced data ( from a producer) by skiping specific key or set of Keys, which could means no data to particular partition.

This answers why leader's outbound traffic reduced (No records for followers to consume)