prometheus rule failing with “many-to-many matching not allowed”

Question

I have an out of the box kube-prometheus v0.3 setup where one of the rules:

alert: KubeletTooManyPods
expr: max
  by(node) (max by(instance) (kubelet_running_pod_count{job="kubelet"}) *
  on(instance) group_left(node) kubelet_node_name{job="kubelet"}) / max by(node)
  (kube_node_status_capacity_pods{job="kube-state-metrics"}) > 0.95

is triggering the following error:

found duplicate series for the match group {instance="10.0.1.151:10250"} on the right hand-side of the operation: [{name="kubelet_node_name", endpoint="https-metrics", instance="10.0.1.151:10250", job="kubelet", metrics_path="/metrics", namespace="kube-system", node="ip-10-0-1-151.eu-west-1.compute.internal", service="prometheus-operator-kubelet"}, {name="kubelet_node_name", endpoint="https-metrics", instance="10.0.1.151:10250", job="kubelet", metrics_path="/metrics", namespace="kube-system", node="ip-10-0-1-151.eu-west-1.compute.internal", service="kubelet"}];many-to-many matching not allowed: matching labels must be unique on one side

What is wrong with this rule?

Denis Popov Denis Popov · Accepted Answer · 2020-11-18T07:18:36

You should inspect services in kube-system namespace, there can be duplicated services, "prometheus-operator-kubelet" and "kubelet" in your case.

prometheus rule failing with “many-to-many matching not allowed”

1 Answers