Stacking/Overlapping metrics by name

Question

I have an application shipping metrics to prometheus over micrometer-jmx and I cannot change the application to use micrometer-prometheus instead. All parameterized metrics are therefore not prometheus labels but are instead encoded directly into the name of the metric.

i.e. instead of requests_Count{processor="BILLING_PROCESSOR", type="SCRIPT"} metrics are in the form of requests_PRC_BILLING_PROCESSOR_TYP_SCRIPT_Count.

Now let's say I want a graph in grafana of request counts grouped (stacked/overlapped) by type. Is there any way I can accomplish that without labels and with metrics in that format? I've managed to construct grafana variables which extract the processor and type values from the metric name but I can't seem to do much with those values.

ahus1 ahus1 · Accepted Answer · 2018-09-15T11:16:02

You could configure Prometheus to convert the metrics names. This is part of relabel-ing available in Prometheus. It is described in the Prometheus Configuration and in a blog post by one of the core contributors.

As extracted from the blog post a metrics can be converted from

memory_pools_PS_Eden_Space_committed

to

memory_pools_committed_bytes{pool="PS_Eden_Space"}

by applying a configuration as follows:

scrape_configs:
  job_name: my_job
  # Usual fields go here to specify targets.
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: '(memory_pools)_(.*)_(\w+)'
    replacement: '${2}'
    target_label: pool
  - source_labels: [__name__]
    regex: '(memory_pools)_(.*)_(\w+)'
    replacement: '${1}_${3}_bytes'
    target_label: __name__

Stacking/Overlapping metrics by name

1 Answers