Prometheus: grouping metrics by metric names

23
votes

Is there a way to group all metrics of an app by metric names? A portion from a query listing all metrics for an app (i.e. {app="bar"}) :

ch_qos_logback_core_Appender_all_total{affiliation="foo",app="bar", instance="baz-3-dasp",job="kubernetes-service-endpoints",kubernetes_name="bar",kubernetes_namespace="foobarz",kubernetes_node="mypaas-dev-node3.fud.com",updatedBy="janedoe"}   44
ch_qos_logback_core_Appender_debug_total{affiliation="foo",app="bar", instance="baz-3-dasp",job="kubernetes-service-endpoints",kubernetes_name="bar",kubernetes_namespace="foobarz",kubernetes_node="mypaas-dev-node23.fud.com",updatedBy="deppba"} 32

I have also tried to use wildcard in the metric name, prometheus is complaining about that. Looking at the metrics, I can see that some of them have dynamic names, most probably delivered by dropwizard metrics. What I ultimately want is a list of all available metrics.

3
prometheus
What do you mean by grouping? What are you ultimately trying to do? – brian-brazil
what i mean is that I am after finding all available metrics. – naimdjon

3 Answers

37
votes

I got a hint from an answer of @brian-brazil, and came to a solution. The following query lists all available metrics:

sum by(__name__)({app="bar"})

Where bar is the application name, as you can see in the log entries posted in the question.

21
votes

{__name__=~".+"} will return all non-stale time series, however this is an expensive query and should be generally avoided.

17
votes

Using directly 

{__name__=~".+"}

will return success but nothing else (too big).

or 

{__name__=~".*"}

while will give us error instead as expected:

parse error at char 17: vector selector must contain at least one non-empty matcher

So my trick here is combining the solutions of brian-brazil and naimdjon using things like

sum({__name__=~"c.*|e.*|n.*|p.*|r.*|k.*|z.*|r.*"}) by (__name__)

  1. since I know the possible prefixes, so I add them to query to make sure at lest it will return something; and further,

  2. to avoid useless fields returned (stressing out the Prometheus), I will only require __name__ in this way by(__name__), all the names of the metrics I need returned as expected.

Actually there is an API to get all the available metric names as:

/api/v1/label/__name__/values