Can't find Flink Histogram metric in Datadog

Question

I integrated Flink job with Datadog. In my Flink job, I added metrics of counter and histogram, but from Datadog side, I could only found the metrics of counter, not histogram.

I'm able to see the histogram metric from Flink side:

but can't find it from Datadog side:

Also I'm able to find the counter metrics I added for the job.

My understanding is

I could find histogram metric in the Flink UI, which means there's no issue of my code that collect metrics
I could find counter metric(from the same job) in Datadog, which means there's no issue of my Flink <--> Datadog integration.

Combine 1) and 2), I can't figure out how to debug it. Any idea? Thanks!

Here's how I created histogram

@transient private var eventTimeLagHistogram: Histogram = _
            
override def open(config: Configuration): Unit = {
  val dropwizardHistogram: com.codahale.metrics.Histogram  =
                  new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500))
            
  eventTimeLagHistogram = getRuntimeContext()
                 .getMetricGroup.addGroup("OrderItemUpdateJobTest")
                  .histogram("eventTimeLagHistogram", new DropwizardHistogramWrapper(dropwizardHistogram))
              }
        
        
override def map(t: ObjectNode): OrderItemUpdate = {
              .....
  eventTimeLagHistogram.update(System.currentTimeMillis()- ItemTimestamp)
              .....
    }

David Anderson David Anderson · Accepted Answer · 2021-09-02T18:59:45

Support for histogram metrics in the datadog metrics reporter was added in Flink 1.12.3 and 1.13.0. See FLINK-20533.

Can't find Flink Histogram metric in Datadog

1 Answers