Spring Cloud Stream Kafka Binder Health Indicator is Failing when Auto-commit is true

Question

We are building the custom DB Sink with Spring cloud stream using Kafka binders. We are using following versions of Spring Cloud and Spring Boot -

<spring-cloud.version>Hoxton.SR11</spring-cloud.version>
<spring.boot.version>2.3.4.RELEASE</spring.boot.version>

Spring cloud stream Kafka binder dependency -

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>

The Sink works perfectly fine, but actuator health endpoint is always DOWN.

After debugging the KafkaBinderHealthIndicator flow for health check, the root cause for this failure is the Kafka consumer that is created with empty client ID and group ID. And with auto commit true for the Sink, Kafka consumer creation is failing with following Error -

org.apache.kafka.common.errors.InvalidConfigurationException: enable.auto.commit cannot be set to true when default group id (null) is used.

We have to use /actuator/health as app health indicator, this has become show stopper.

Please let us know which versions of Spring cloud and Spring boot can work for binder health.

Gary Russell Gary Russell · Accepted Answer · 2021-04-30T12:45:58

You should not use enable.auto.commit=true with SCSt - the framework will automatically take care of committing the offsets when the listener returns normally.

This is much more deterministic than Apache Kafka's auto commit mechanism.

Spring Cloud Stream Kafka Binder Health Indicator is Failing when Auto-commit is true

1 Answers