Kafka consumers all have assigned_partition set to 0 but at runtime are assigned differnt partitions

Question

I am not sure if i am doing something wrong - but if someone can please help shed some light on this.

Scenario:

Launching a broker having 1 topic with two partitions.
Launching 2 consumers - with two separate client_id and both registering to the same consumer group and listening to the same topic (above).

Before i fire up the producer - i wanted to see what partitions these consumers are assigned to and see :

Listening on Topic : TOPIC_2PART
Status :True
ClientId : consumer_3_1
***************
  CONFIG 
***************
bootstrap_servers -> $node1_port
client_id -> consumer_3_1
group_id -> CGRP_TOPIC_2PART
value_deserializer -> <function CreateConsumer.consume.<locals>.<lambda> at 0x0000020175217DC8>
partition_assignment_strategy -> (<class 'kafka.coordinator.assignors.range.RangePartitionAssignor'>, <class 'kafka.coordinator.assignors.roundrobin.RoundRobinPartitionAssignor'>)
metric_group_prefix -> consumer
***************
  METRICS 
***************         
    consumer-fetch-manager-metrics
    **assigned-partitions->0.0**

And for the second consumer :

Listening on Topic : TOPIC_2PART
Status :True
ClientId : consumer_3
***************
  CONFIG 
***************
bootstrap_servers -> $node1_port
client_id -> consumer_3
group_id -> CGRP_TOPIC_2PART
value_deserializer -> <function CreateConsumer.consume.<locals>.<lambda> at 0x00000118DC956DC8>
partition_assignment_strategy -> (<class 'kafka.coordinator.assignors.range.RangePartitionAssignor'>, <class 'kafka.coordinator.assignors.roundrobin.RoundRobinPartitionAssignor'>)
selector -> <class 'selectors.SelectSelector'>
***************
  METRICS 
***************
            kafka-metrics-count
            consumer-fetch-manager-metrics
            consumer-coordinator-metrics
    assigned-partitions->0.0

So above before either consumer starts consume assigned-partitions are 0

After producer starts sending messages to both partitions on the topic - i see one consumer having partition 0 and the other partition 1

ConsumerRecord(topic='TOPIC_2PART', **partition=0**, offset=14, timestamp=1616346804116, timestamp_type=0, key=None, value={xxxx}, headers=[], checksum=None, serialized_key_size=-1, serialized_value_size=99, serialized_header_size=-1)

And the second consumer having

ConsumerRecord(topic='TOPIC_2PART', **partition=1**, offset=4, timestamp=1616346723367, timestamp_type=0, key=None, value={}, headers=[], checksum=None, serialized_key_size=-1, serialized_value_size=83, serialized_header_size=-1)
ConsumerRecord(topic='TOPIC_2PART', **partition=1**, offset=5, timestamp=1616346743556, timestamp_type=0, key=None, value={}, headers=[], checksum=None, serialized_key_size=-1, serialized_value_size=94, serialized_header_size=-1)

My question is why am i seeing the assigned-partition on both consumers set to 0 ...

Thanks

Lior Chaga Lior Chaga · Accepted Answer · 2021-03-22T06:36:29

The assigned-partitions metric is the number of partitions assigned to each consumer, not the partitions ids. With two consumers and two partitions, this should be set to 1 for each of them once you subscribe. So question is - when does subscription occur? Seems like the information you've pasted was extracted before the subscription.

Kafka consumers all have assigned_partition set to 0 but at runtime are assigned differnt partitions

1 Answers