Kafka consumer & Partition query

0
votes

I am new to Kafka and read few tutorials. I couldn't understand the relationship between consumer and partition.

Please address my below queries.

  1. As per documentation, only one consumer can consume message in group. Why do we need to create more consumers in that same group? What is the benefit?

  2. Does consumer are assigned to individual partition by ZK? , if Yes, if producer sends message to different partition then how will other partition’s consumer consume the message ?

  3. I have one topic and that has 3 partitions. I post msg, it goes to P0. I have 5 consumers (different consumer group). Will all consumers read message from P0? if I increase many Consumer, will all read message from same P0 ? If all consumer read from same PO then how performance will be high?

  4. How rebalancing is working? will it work when you increase consumer group or consumer in same group ?

Please clarify my questions and give some example.

1
apache-kafkakafka-consumer-api
1. More consumers - more messages handled in parallel . Each message handled by one consumer inside group – MGolovanov
2 Consumer could assigned to individual partition explicitly using consumer API, but could not. By default consumer readings balanced by kafka broker – MGolovanov
1. more messages handled in parallel . - will all consumer consume message from same topic to handle in parallel. ?. I hope only one consumer will read message from one topic. rest all consumer on the same group cannot get message from same topic. if Yes, then how parallel – Gnana
Right - all consumers consume message from same topic to handle in parallel. Wrong - I hope only one consumer will read message from one topic. – MGolovanov
Please check my below answer and confirm. Thanks. – Gnana

1 Answers

1
votes

  1. Yes, only once consumer in consumer group can consume message from one partition, rest of consumer in the same group will be assigned to remaining partition to do parallel process. Advantage is parallel processing.

  2. Yes partition will be assigned to consumer by ZK. Based on partition count and consumer count, allocation will be done. Ex: Topic (Test) has 3 Partition (P1, P2, and P3). We have one consumer (C1). C1 will read message from all partition. If you add one more consumer in that same group (c2). ZK will assign P1, p2 to C1 and P3 goes to C2. If add one more consumer (C3) than P1=C1, P2=C2 and P3=C3. No of consumer should not be greater than no of partition for that topic.

  3. Above point will answer this one.

  4. Rebalancing will work when you add consumer on the same consumer group.