Load balance Kafka record consumption when using topic pattern

Question

If two processes which uses the same consumer group (group.id) subscribe to the same topicPattern - will Kafka than load balance the resolved topics between the two processes?

In my case about 400 topics match the pattern and each topic has one partion. I wonder if I can load balance the consumption of the topics via different processes with the same consumer group but it seems that only one process handles all the topics. Is this because only partitions are load balanced - or maybe because a topic pattern is used by the subscription?

I made some deeper debugging. My test setup is not really two processes but two threads - I guess that should not matter. Debug printing the topic description outputs something like this (IPs are obfuscated):

(groupId=foo-bar-group, isSimpleConsumerGroup=false, 
  members=
   (memberId=KafkaTestApplication:KafkaTestApplication-1-20126d55-678d-46f5-bc73-3769db2c8901, 
     clientId=KafkaTestApplication:KafkaTestApplication-1, 
     host=/aa.bb.c.dd, 
     assignment=(topicPartitions= <all topics here>)),
   (memberId=KafkaTestApplication:KafkaTestApplication-2-8d34e81a-8f57-4d4e-bd9e-b56edc4e706c, 
     clientId=KafkaTestApplication:KafkaTestApplication-2,
     host=/aa.bb.c.dd, 
     assignment=(topicPartitions=)), 
     partitionAssignor=range, 
     state=Stable, 
     coordinator=aaa.bb.ccc.d:9092 (id: 9 rack: null)
)

I can see that I have two different members in the consumer group with two different clientIds. Nevertheless all topics are assigned to the first consumer and the second consumer has no topics assigned. I expected that both members get some topics. Both members poll every few seconds.

How are you checking "it seems" this way? And do you really want 200 partitions per consumer process? Seems like a lot — OneCricketeer
For now I only checked by inspecting the logs :-| - In case rebalancing works I would probably start as much processes as needed to handle the expected amount of records. — FrVaBe

OneCricketeer OneCricketeer · Accepted Answer · 2019-02-08T14:56:42

When the second process of the same group starts, the group should rebalance the Topic partition assignment, yes.

However, I've not had any experience with subscription patterns, but I don't expect it to work differently than providing a direct list of topics

Load balance Kafka record consumption when using topic pattern

2 Answers