I am attempting to send data to Prometheus from my Spring Boot app using Micrometer's @Timed annotation and am seeing strange behavior. I see my method's data in prometheus, but I'm also seeing it fail in some instances with the exception below. I am including the necessary libraries and have followed Spring Boot configuration instructions for adding Micrometer @Timed annotation. Feels like a Spring Boot / Micrometer bug when constructing the keys for the metric. My timed annotation is very simple:
@GetMapping(value = {"/pageable"}, produces = MediaType.APPLICATION_JSON_VALUE)
@Timed("employees.get.pageable")
public Page<Employee> all(@SortDefault(sort = "id", direction = Sort.Direction.ASC) Pageable p) {
return (Page<Employee>) employeeRepository.findAll(p);
}
Any ideas what I might be doing wrong? I'll be glad to respond with additional config, but didn't want to paste my entire project here. The stacktrace:
2020-01-21 18:53:40.559 ERROR 1 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception
java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'companies_get_single_seconds' containing tag keys [class, exception, method]. The meter you are attempting to register has keys [exception, method, outcome, status, uri].
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$collectorByName$9(PrometheusMeterRegistry.java:382) ~[micrometer-registry-prometheus-1.3.2.jar!/:1.3.2]
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1932) ~[na:na]
at io.micrometer.prometheus.PrometheusMeterRegistry.collectorByName(PrometheusMeterRegistry.java:369) ~[micrometer-registry-prometheus-1.3.2.jar!/:1.3.2]
at io.micrometer.prometheus.PrometheusMeterRegistry.newTimer(PrometheusMeterRegistry.java:175) ~[micrometer-registry-prometheus-1.3.2.jar!/:1.3.2]
at io.micrometer.core.instrument.MeterRegistry.lambda$timer$2(MeterRegistry.java:270) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:575) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:528) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:268) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.Timer$Builder.register(Timer.java:464) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.composite.CompositeTimer.registerNewMeter(CompositeTimer.java:140) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.composite.CompositeTimer.registerNewMeter(CompositeTimer.java:31) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.composite.AbstractCompositeMeter.add(AbstractCompositeMeter.java:66) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at java.base/java.util.Collections$SetFromMap.forEach(Collections.java:5581) ~[na:na]
at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$null$0(CompositeMeterRegistry.java:65) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lock(CompositeMeterRegistry.java:184) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$new$1(CompositeMeterRegistry.java:65) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:585) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:528) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:268) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at io.micrometer.core.instrument.Timer$Builder.register(Timer.java:464) ~[micrometer-core-1.1.7.jar!/:1.1.7]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.stop(WebMvcMetricsFilter.java:180) ~[spring-boot-actuator-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.record(WebMvcMetricsFilter.java:174) ~[spring-boot-actuator-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:130) ~[spring-boot-actuator-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104) ~[spring-boot-actuator-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.1.10.RELEASE.jar!/:5.1.10.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.1.10.RELEASE.jar!/:5.1.10.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.1.10.RELEASE.jar!/:5.1.10.RELEASE]
