Kafka Streams: Store state during rebalance

Question

While in the middle of traversing through the iterator in the periodic execution of punctuate method, if a rebalance occurs would the below loop stop OR throw an exception? Or would it resume automatically, if so would it resume with the same partition shard?

    KeyValue<String, House> curr;
    KeyValueIterator<String, House> houseIterator = houseStore.all();
    while (houseIterator.hasNext()) {
        j++;
        curr = houseIterator.next();
        houseStore.delete(curr.key);
    }
    houseIterator.close();

Matthias J. Sax Matthias J. Sax · Accepted Answer · 2018-06-10T07:29:25

The executing thread will not call KafkaConsumer#poll() until punctuate() is finished. Thus, the thread (ie, punctuate) will continue the run.

If punctuate() finishes before max.poll.interval.ms passed, it will rebalance correctly. If punctuate finished after max.poll.interval.ms the consumer would drop out of the consumer group, and it partitions might get reassigned. However, depending on your code within punctuate, it would not be noticed and punctuate might still finish. Later, when poll is called, the instance would rejoin the group via another rebalance.

Note though, that it is recommended to configure max.poll.interval.ms such that the instances does not drop out of the consumer group in the first place. Ie, the value should be set to a larger value than the expected runtime of punctuate.

Kafka Streams: Store state during rebalance

1 Answers