Will the KafkaConsumer still continue to fetch in the background for topic partitions that are in the paused state?

Question

I'm looking at the KafkaConsumer source code here. From looking at the source code (which is in java, which I'm not too familiar with), pausing a topicpartition seems to set its offset state to paused. I'm trying to figure out what happens the to fetches that the KafkaConsumer does in the background and whether pausing the topic partition stops the prefetching that the kafka consumer does, or whether it continues to prefetch but it doesn't return ConsumerRecords for that topic partition on poll.

Not sure about the prefetching, but did you read the comment? Future calls to {@link #poll(Duration)} will not return any records from these partitions until they have been resumed using {@link #resume(Collection)} — OneCricketeer
Yes, I read that. I understand that the calls to poll won't return any records. I'm not sure if that means that it will also stop the background prefetching. — adityagnet
I would assume that whatever had been prefetched before the pause will be returned, but the subsequent poll call will return null — OneCricketeer

amethystic amethystic · Accepted Answer · 2018-10-22T08:16:43

Both. Pausing a partition stops the prefetching for that partition and also returns empty on poll.

Will the KafkaConsumer still continue to fetch in the background for topic partitions that are in the paused state?

1 Answers