Storm kafka warning message there are more tasks than partitions some tasks will be idle

Question

I have storm topology which use KafkaSpout to receive message from Kafka broker. But i don't see the spout processing any messages. In the worker logs i found the below snippet. Storm version-0.9.3

2017-12-13T18:43:49.598+0000 s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager connections
2017-12-13T18:43:49.600+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}}
2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle
2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] Task [1/2] no partitions assigned
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] Deleted partition managers: []
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] New partition managers: []
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] Finished refreshing
2017-12-13T18:43:50.111+0000 s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager connections
2017-12-13T18:43:50.113+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}}
2017-12-13T18:43:50.113+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle
2017-12-13T18:43:50.113+0000 s.k.KafkaUtils [WARN] Task [1/2] no partitions assigned
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] Deleted partition managers: []
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] New partition managers: []
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] Finished refreshing

I am not sure though why i am getting such warning messages. I tried to verify the kafka topic using the kafka-topic cli but i found 3 partitions.

./kafka-topics.sh  --zookeeper xxx.xxx.xxx.xxx:2181 --topic topic-name  --describe
Topic:topic-name PartitionCount:3  ReplicationFactor:1 Configs:
  Topic: topic-name  Partition: 0  Leader: 2 Replicas: 2 Isr: 2
  Topic: topic-name  Partition: 1  Leader: 2 Replicas: 2 Isr: 2
  Topic: topic-name  Partition: 2  Leader: 2 Replicas: 2 Isr: 2

What am i doing wrong here or what could be the possible issue of not receiving the messages ?

Stig Rohde Døssing Stig Rohde Døssing · Accepted Answer · 2017-12-30T18:39:16

Note the lines

2017-12-13T18:43:49.600+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}} 2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle

It's failing to read the list of partitions from Zookeeper. When you use the storm-kafka KafkaSpout, you need to configure it to connect to your Kafka cluster's Zookeeper. You should be setting it in the SpoutConfig via https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/ZkHosts.java#L33. The zookeeper connection string syntax is the same as what you're passing to kafka-topics.sh in the terminal.

Storm kafka warning message there are more tasks than partitions some tasks will be idle

1 Answers