Kafka, getting the partition from consumerRecord

0
votes

If I have one topic which has 5 partitions and then I have a service consuming off these 5 partitions. Then at the consumer I poll and get back an array of ConsumerRecords.

Can each individual ConsumerRecord be from any of those 5 partitions?

I see ConsumerRecord has a method:

int partition() { return this.partition; }

So will the array of ConsumerRecords on each poll be made of records which can come from any of the 5 partitions?

If there was a second service consuming from the same 5 partitions, so then in total 2 services consuming from the same 5 partitions. Will the partition number be the same in each service for the same Kafka message?

Thanks.

1
apache-kafkakafka-consumer-api

1 Answers

1
votes

So will the array of ConsumerRecords on each poll be made of records which can come from any of the 5 partitions?

Yes, when only one service is running, it will consume from all 5 partitions. So, array of consumerRecords on each poll will be made of records from any of 5 partitions.

If there was a second service consuming from the same 5 partitions, so then in total 2 services consuming from the same 5 partitions. Will the partition number be the same in each service for the same Kafka message?

If 2 instances of service is running, since both instance are in same consumer group, they will consume from separate partitions(e.g. 1st instance will consume from 1st and 2nd partition. 2nd instance will consume from 3rd, 4th, 5th partition). So, partition number of message in both instance will not be same.

Consumers label themselves with a consumer group name(consumer.id), and each record published to a topic is delivered to one consumer instance within each subscribing consumer group.

In your case, you have 2 consumers(2 instances) belonging to same Consumer group, so each Instances will consume from different partitions so that both consumers collectively get data from all partitions.

If both services are in different consumer groups.

In this case, partition number will be the same in each service for the same Kafka message.