Kafka Consumer metrics gone upon upgrade from Spring Boot 2.2.2 to 2.3.0

Question

Problem:

We upgraded our Spring Boot version from 2.2.2 to 2.3.0, and all of the kafka_consumer_* metrics that were seen in the Prometheus endpoint in 2.2.2 are not visible in 2.3.0.

For example, all of the below are missing:

kafka_consumer_records_consumed_total_records_total
kafka_consumer_records_lag_records
kafka_consumer_fetch_latency_max_seconds
kafka_consumer_bytes_consumed_total_bytes_total

Not sure if we're missing some kind of configuration or something buried in the docs...

What has been tried:

Combed the Spring Boot 2.3.0 release notes, updated micrometer documentation, and updated spring-kafka documentation for why this might be happening
Googled to what feels like the ends of the Earth
Tried upgrading to Spring Boot 2.2.7 and the kafka metrics are still there, only upgrading to 2.3.0 seems to cause the issue
Removed any non-needed dependency/customization that was in our code for the project, and bare bones just connected to a kafka container on the localhost, and the metrics still don't appear

Relevant code/details:

We're using Red Hat AMQ Streams for our kafka broker (kafka version 2.1.1)
The only thing we've changed in our environment is the Spring Boot version (and dependencies that are automatically pulled in/updated) to recreate this issue

Below is our build.gradle.kts before the change:

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
    id("org.springframework.boot") version "2.2.2.RELEASE"
    id("io.spring.dependency-management") version "1.0.9.RELEASE"
    kotlin("jvm") version "1.3.72"
    kotlin("plugin.spring") version "1.3.72"
}

group = "ourGroup"
version = "0.0.1"
java.sourceCompatibility = JavaVersion.VERSION_1_8

repositories {
    mavenCentral()
}

extra["springCloudVersion"] = "Hoxton.RELEASE"

dependencyManagement {
    imports {
        mavenBom("org.springframework.cloud:spring-cloud-dependencies:${property("springCloudVersion")}")
    }
}

dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-stream-kafka")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("io.micrometer:micrometer-registry-prometheus")
    implementation("org.springframework.boot:spring-boot-starter-webflux")
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
    testImplementation("org.springframework.boot:spring-boot-starter-test") {
        exclude(group = "org.junit.vintage", module = "junit-vintage-engine")
    }
    testImplementation("io.projectreactor:reactor-test")
    testImplementation("org.springframework.security:spring-security-test")
}

tasks.withType<Test> {
    useJUnitPlatform()
}

tasks.withType<KotlinCompile> {
    kotlinOptions {
        freeCompilerArgs = listOf("-Xjsr305=strict")
        jvmTarget = "1.8"
    }
}

Now, if we just update our build.gradle.kts with a new spring boot version like in the below line our kafka metrics disappear:

    id("org.springframework.boot") version "2.3.0.RELEASE"

Below are screenshots of the prometheus metrics we were seeing before and after the change:

Before 2.3.0 upgrade

After 2.3.0 upgrade

Thanks in advance for the help! Let me know if you need any other details!

I am using spring-kafka and just tested @AlanYeung's issue. I can confirm that in Spring Boot 2.3.0.RELEASE, the kafka_consumer_* metrics are gone in /actuator/prometheus. I can still see them in 2.2.7.RELEASE. — jumping_monkey
Kafka meters were migrated from scraping JMX MBeans to using native metrics in Boot 2.0. See this commit and this one. The last one uses this. I'll see if I can figure out what's broken tomorrow. — Gary Russell
Awesome @GaryRussell, thanks a lot for the pointers! I have the kafka_consumer_* metrics back in 2.3.0.RELEASE. Cheers! — jumping_monkey
I'd like to see more new contributors put this much effort into their first question. Nice job, Alan! — Jeremy Caney

Gary Russell Gary Russell · Accepted Answer · 2020-06-10T13:51:06

It works for normal spring-kafka consumers

@SpringBootApplication
public class So62292889Application {

    public static void main(String[] args) {
        SpringApplication.run(So62292889Application.class, args);
    }

    @KafkaListener(id = "so62292889", topics = "so62292889")
    public void listen(String in) {
        System.out.println(in);
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("so62292889").partitions(1).replicas(1).build();
    }

    @Bean
    public ApplicationRunner runner(MeterRegistry registry) {
        return args -> {
            registry.getMeters().forEach(meter -> System.out.println(meter.getId()));
        };
    }

}

MeterId{name='kafka.consumer.outgoing.byte.total'...

I see you are using spring-cloud-stream. The problem is that this project creates its own producer/consumer factories and doesn't add the Micrometer listeners.

I opened an issue against the binder.

Kafka Consumer metrics gone upon upgrade from Spring Boot 2.2.2 to 2.3.0

Problem:

What has been tried:

Relevant code/details:

2 Answers