Prometheus filtering based on Labels

Question

How to add label filters in Prometheus query?

kube_pod_info

kube_pod_info{created_by_kind="ReplicaSet",created_by_name="alertmanager-6d9f74d4c5",instance="kube-state-metrics:8080",job="kube-state-metrics",namespace=“test",pod="alertmanager-6d9f74d4c5-xlqrv"}

kube_pod_labels

kube_pod_labels{instance="kube-state-metrics:8080",job="kube-state-metrics",label_app="alertmanager",label_pod_template_hash="6d9f74d4c5",namespace=“test",pod="alertmanager-6d9f74d4c5-xlqrv”,label_source=“k8s"}

Here, I have kube state metrics info in prometheus for kube_pod_info & kube_pod_labels.

kube_pod_info{namespace="test"} ---> Filters pods by namespace test.

Here, I want to include filter based on labels as well. I have a label called "label_source=“k8s" in kube_pod_labels. How can I join kube_pod_info & kube_pod_labels to apply label filter as well?

Answer 1

You can use + operator to join metrics. Here, group_left() will include the extra label: label_source from the right metric kube_pod_labels. The metric you're joining is forced to zero ( i.e. 0 * kube_pod_labels ) so that it doesn't affect the result of first metric.

(
kube_pod_info{namespace="test"}
)
   + on(namespace) group_left(label_source)
(
   0 * kube_pod_labels
)

Answer 2

As per the "Join Metrics" documentation here, you can essentially combine the labels from the kube_pod_info query with those of the kube_pod_labels query.

Something along the lines of this:

kube_pod_info{namespace="test"} * on (namespace, pod) group_left() kube_pod_labels{label_source="k8s"}

This query should give you all of the pods in the namespace test, which have the source label set to k8s.