Kafka - Partition Rebalance and Data Consistency

Question

I am using Kafka 0.8.1.
I am using Kafka to distribute our events across our processing cluster.
I am using 1 topic with multiple partitions.
My concern is regarding the Kafka rebalancing process. Currently our processing logic occurs on the machine running the Kafka consumer so I am wondering what will happen in case of rebalance?
What is the best practice to avoid race between the new node handling the partition and the old one?
The processing result is eventually persisted in to some storage and I am wondering how to avoid old consumer overriding new consumer updates.

In case I use auto.commit false, does Kafka have some guarantee not to rebalance when there are uncommitted messages.

Thanks.

Answer 1

we have this considering either, i would say the best reference would definitely be the source code of Kafka.

Regrading to your question specifically, when rebalance happened, kafka will first close all the fetchers including clear uncommitted messages in the blocking queue, and then swith. Thus there exist the timing when Consumer A (old) and Consumer B (new) consume the message from the same partition.