What does KafkaConsumer.commitSync() actually commit?

Question

Does KafkaConsumer.commitSync just commit "offsets returned on the last poll()" as JavaDoc claims, (which may miss some partitions not included in the last poll result), or it is actually committing the latest positions for all subscribed partitions? Asking because the code suggests the latter, considering allConsumed: https://github.com/apache/kafka/blob/2.4.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1387

@Override
public void commitSync(Duration timeout) {
    acquireAndEnsureOpen();
    try {
        maybeThrowInvalidGroupIdException();
        if (!coordinator.commitOffsetsSync(subscriptions.allConsumed(), time.timer(timeout))) {
            throw new TimeoutException("Timeout of " + timeout.toMillis() + "ms expired before successfully " +
                    "committing the current consumed offsets");
        }
    } finally {
        release();
    }
}

Michael Heil Michael Heil · Accepted Answer · 2020-04-28T16:41:25

It only commits the offsets that were actually polled and processed. If some offsets were not included in the last poll, then those offsets will not be committed.

It will not commit the latest positions for all subscribed partitions. This would interfere with the Consumer Offset management concept of Kafka to be able to re-start an application where it left off.

From my understanding, the allConsumed is equivalent to all offsets included in the last poll which the comment of the commitSync also documents

Commit offsets returned on the last poll() for all the subscribed list of topics and partitions.

What does KafkaConsumer.commitSync() actually commit?

1 Answers