python-kafka consumer.assign and consumer.subscribe mixed in a same group cause crash

1
votes

OS:mac Darwin Kernel Version 15.6.0

Kafka version:0.11.0.0

  1. Use python-kafka to create a consumer, group_id = 'my_group', use the assign method to subscribe to a specific topic = 'assign_topic', partition = [0,1,2,3,4], create five consumer to consume messages, consumer.assign() do not use consumer's group Management functionality.

  2. Now I create a new consumer, subscribe the new topic = 'subscribe_topic' using consumer.subscribe() method, the partitions can be dynamically balanced, partitions = [0,1,2,3,4], group_id = 'my_group', start the consumer.

  3. Results: the consumer created in Step2 is running successfully with the topic of 'subscribe_topic', but the five consumers created in Step1 looks like died (the process did not throw an exception, use ps -ef can find pid is still alive), but cannot get any messages.Kafka official websitesays： It is not possible to mix both answer to specific partitions (with no load balancing) and to topics (with load balancing) using the same consumer instanc e. But the key point is that my mixed subscription is not used in a same consumer instance.

  4. When I changed the consumer group_id in Step2, such as group_id='hello_topic', everything goes well. I am not sure what reason cause the issue. Maybe I use the consumer incorrectly. Maybe...

1
pythonapache-kafka

1 Answers

0
votes

Why your case happens ?

Because in the step2, a consumer whose group is same as step1's consumers (my_group) use subscribe() method to subscribe a new topic.

Subscribe to a list of topics, or a topic regex pattern.

Partitions will be dynamically assigned via a group coordinator. Topic subscriptions are not incremental: this list will replace the current assignment (if there is one).

Notice this method will replace older assignment, a new topic list will replace the old topic list, so It's action is that the consumer group "my_group" unsubscribe the "assign_topic" and subscribe the "subscribe_topic".

A thing you should know is that Only the consumer group can be topic subscriber and the consumer is actually a splitting process of this group. So other consumers in "my_group" can not get any message from older topic.

And this is also because your consumers in step1 use the manual partition subscription So it can not listen to this group topic change thing. If you use the subscribe() method in step1. Then the result will be different.