The docs are great about explaining how to set a taint on a node, or remove one. And I can use kubectl describe node
to get a verbose description of one node, including its taints. But what if I've forgotten the name of the taint I created, or which nodes I set it on? Can I list all of my nodes, with any taints that exist on them?
12 Answers
In Kubernetes 1.6.x the node taints have moved into the spec. Therefore the above answer by jaxxstorm will not work. Instead, you can use the following template.
{{printf "%-50s %-12s\n" "Node" "Taint"}}
{{- range .items}}
{{- if $taint := (index .spec "taints") }}
{{- .metadata.name }}{{ "\t" }}
{{- range $taint }}
{{- .key }}={{ .value }}:{{ .effect }}{{ "\t" }}
{{- end }}
{{- "\n" }}
{{- end}}
{{- end}}
I have that saved into a file and then reference it like so:
kubectl get nodes -o go-template-file="./nodes-taints.tmpl"
You'll get output like so:
Node Taint
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=etcd:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=jenkins:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=etcd:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=containerlinux-canary-channel-workers:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=jenkins:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=etcd:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=etcd:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=etcd:NoSchedule
ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal dedicate=jenkins:NoSchedule
I'm not a huge go template user so I'm sure there are some things I could have done better but it is what it is.
Same as above but all in one line:
kubectl get nodes -o go-template='{{printf "%-50s %-12s\n" "Node" "Taint"}}{{- range .items}}{{- if $taint := (index .spec "taints") }}{{- .metadata.name }}{{ "\t" }}{{- range $taint }}{{- .key }}={{ .value }}:{{ .effect }}{{ "\t" }}{{- end }}{{- "\n" }}{{- end}}{{- end}}'
The simplest way to do this without using any extra tools such as JQ is to use the custom-columns output option.
$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints --no-headers
Output:
master-11 [map[effect:PreferNoSchedule key:node-role.kubernetes.io/master]]
master-12 [map[effect:PreferNoSchedule key:node-role.kubernetes.io/master]]
master-13 [map[effect:PreferNoSchedule key:node-role.kubernetes.io/master]]
With something like Taints where it is a map or list and you want it to look clean for parsing with some other tool you can clean them up using you can use something similar to the answer by Edwin Tai but with a little extra smarts to extract the keys.
kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints[*].key}{"\n"}{end}'
Output:
master-11 node-role.kubernetes.io/master
master-12 node-role.kubernetes.io/master
master-13 node-role.kubernetes.io/master
worker-21 thegoldfish.org/storage thegoldfish.org/compute
worker-22 thegoldfish.org/storage thegoldfish.org/compute
worker-23 thegoldfish.org/compute
worker-24 thegoldfish.org/storage thegoldfish.org/compute
Extra examples:
Using this method you can easily create custom outputs
Quick overview of nodes:
kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture,KERNEL:.status.nodeInfo.kernelVersion,KUBLET:.status.nodeInfo.kubeletVersion,CPU:.status.capacity.cpu,RAM:.status.capacity.memory
Output:
NAME ARCH KERNEL KUBLET CPU RAM
master-11 amd64 3.10.0-1062.9.1.el7.x86_64 v1.17.0 6 7910096Ki
master-12 amd64 3.10.0-1062.9.1.el7.x86_64 v1.17.0 6 7910096Ki
master-13 amd64 3.10.0-1062.9.1.el7.x86_64 v1.17.0 6 7910096Ki
Overview of pods and where to find them sorted by creation time:
kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName,HOSTIP:.status.hostIP,PHASE:.status.phase,START_TIME:.metadata.creationTimestamp --sort-by=.metadata.creationTimestamp
Output:
NAMESPACE NAME NODE HOSTIP PHASE START_TIME
kube-system kube-proxy-rhmrz master-11 192.168.121.108 Running 2019-12-26T14:22:03Z
kube-system coredns-6955765f44-777v9 master-11 192.168.121.108 Running 2019-12-26T14:22:03Z
kube-system coredns-6955765f44-w7rch master-11 192.168.121.108 Running 2019-12-26T14:22:03Z
kube-system kube-scheduler-master-11 master-11 192.168.121.108 Running 2019-12-26T14:22:05Z
kube-system kube-controller-manager-master-11 master-11 192.168.121.108 Running 2019-12-26T14:22:05Z
kube-system etcd-master-11 master-11 192.168.121.108 Running 2019-12-26T14:22:05Z
kube-system kube-apiserver-master-11 master-11 192.168.121.108 Running 2019-12-26T14:22:05Z
kube-system calico-node-sxls8 master-11 192.168.121.108 Running 2019-12-26T14:55:41Z
kube-system calico-kube-controllers-6d85fdfbd8-dnpn4 master-11 192.168.121.108 Running 2019-12-26T14:55:41Z
kubernetes-dashboard dashboard-metrics-scraper-76585494d8-jx9cg master-11 192.168.121.108 Running 2019-12-26T16:10:16Z
kubernetes-dashboard kubernetes-dashboard-5996555fd8-5z5p2 master-11 192.168.121.108 Running 2019-12-26T16:10:16Z
The documentation for this is https://kubernetes.io/docs/reference/kubectl/overview/#custom-columns
I was looking to get the list of nodes that have a specific Taint. I only found this SO answer, so if anybody is looking for this answer, here is the solution:
kubectl get nodes -o go-template='{{range $item := .items}}{{with $nodename := $item.metadata.name}}{{range $taint := $item.spec.taints}}{{if and (eq $taint.key "node-role.kubernetes.io/master") (eq $taint.effect "NoSchedule")}}{{printf "%s\n" $nodename}}{{end}}{{end}}{{end}}{{end}}'
On my cluster, the output is:
preprod-master
preprod-proxy
#Check Node Taints
kubectl get nodes -o=custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect
Let me try and explain what this first one means and then rest should fall in place:
NodeName:.metadata.name
ColumnName: JSONPATH to the attribute you are looking for.
ColumnName can be anything you want it to be.
Something like NodeName:items[*].metadata.name is equivalent to running $kubectl get nodes -o=jsonpath='{.items[*].metadata.name}' but with custom-columns flag you get values in rows and columns format.
Note: You don't need to start with .items[*]. It already parses that with custom-column flag
so now all the columns explained:
NodeName:.metadata.name - Get Node Names and put it under NodeName Column
TaintKey:.spec.taints[*].key - return all the keys for taints by looking under taints map and put it under TaintKey custom-column
TaintValue:.spec.taints[*].value - same as key but you are returning the value from the taints map.
TaintEffect:.spec.taints[*].effect - same as key but you are returning the effect from the taints map.
You set it under and alias like
alias get-nodetaints="kubectl get nodes -o=custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect"
and you have your own nice command to get all taints and your output should look something like below
You can use kubectl
's go-template output options to help you here,
kubectl get nodes -o go-template='{{range .items}}{{if $x := index .metadata.annotations "scheduler.alpha.kubernetes.io/taints"}}{{with $x := index .metadata.name}}{{.}}{{printf "\n"}}{{end}}{{end}}{{end}}'
On my cluster, this prints my masters, which are tainted:
kubemaster-1.example.net
kubemaster-2.example.net
kubemaster-3.example.net
The CMD kubectl provides an arguments jsonpath to search and format the output after getting. You can check the doc k8s jsonpath for detail.
kubectl get node -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'
For further info, you can check the source code that reflect the source data with the method FindResults
Below commands worked for me:
- If you have the node IP, You can try
kubectl get node $node_ip -o json | jq '.spec.taints'
Output:
[
{
"effect": "NoSchedule",
"key": "dedicated"
}
]
(OR)
kubectl describe node $node_ip | grep -i Taints
Output:
Taints: dedicated:NoSchedule
- To get all Taint config on namespace
kubectl get nodes -o json | jq '.items[].spec.taints'
Output:
[
{
"effect": "NoSchedule",
"key": "dedicated"
}
]