Exclude k8s nodes from grafana monitoring

3
votes

We have a kubernetes cluster made up of 5 nodes. 2 of the nodes are only used for KIAM and the other 3 are for container deployments.

I have prometheus and grafana deployed and configured and I need to configure monitoring for CPU, memory and pod usage. However I want to totally exclude the nodes hosting KIAM from any stats or alert thresholds.

The only thing I can see being returned by prometheus that can identify the nodes i need is label_workload="gp" from the kube_node_labels metric. What I don't know how to do is to get grafana to only use these nodes in it's calculations.

Perhaps it's possible to have some sort of query join or subselect to identify the node names to include??

I'd appreciate any help on this!!!

1
kubernetesgrafanaprometheus
Where are the metrics coming from? Who exports them? prometheus-node-exporter? – Moritz Schmitz v. Hülst
They're coming from the node-exporter deamonset that prometheus is then scraping – Steve

1 Answers

2
votes

I believe the node_uname_info metrics is a better metric to get all your node infos. So I will explain using that metric.

You have two options:

Option 1: You hard-code the nodenames into your Grafana dashboard. Your query should look something like this node_uname_info{nodename=~"node1|node2|node3"} then. node1-3 are the nodes which you want to have the metrics for.

Option 2: You create a variable and allow the user to select the nodes. Let's say the variable name is $nodes and the query should be label_values(node_uname_info, nodename). It should be allowed to select multiple values. Next in your query you can then do node_uname_info{nodename=~"$nodes"} to only show metrics for selected nodes.