Spring Cloud Stream Kafka consumer with time based auto commit

Question

I have a Spring Cloud Stream app using the Kafka binder. It appears to be committing offsets automatically based on AckMode=BATCH (had to find that in debug mode as it doesn't appear documented). This is causing significant overhead to the broker in commit frequency.

I would like to either switch to using the native Kafka auto commit, e.g.:

enable.auto.commit: true
auto.commit.interval.ms: 5000

Or switch to using TIME config with 5000 ms ackTime as documented here:

https://docs.spring.io/spring-kafka/reference/htmlsingle/#committing-offsets

Is this possible using spring-cloud-stream-binder-kafka? Can you provide a sample config? Native configs above appear to be ignored (in configuration block).

Answer 1

The AckMode is ignored if enable.auto.commit is true. Bear in mind, however, that, unless you are using the 0.10.1.0 client or above, the broker will perform a rebalance if your consumer is slow since, to support this mode, we have to invoke your listener on the consumer thread. See KIP-62.

Since (the upcoming) version 1.3, the listener is always invoked on the consumer thread because the rebalance threat is eliminated (1.3 and above use the 0.11 client).

You can also use AckMode.TIME, AckMode.COUNT or AckMode.COUNT_TIME for even more flexibility over commit intervals. You might find the COUNT_TIME provides the best of both worlds.

Bear in mind, the risk of redelivery after a failure is increased with time-based (or high count-based) ack strategies.