Kubernetes V1.16.8 doesn't support 'node-role' label using “--node-labels=node-role.kubernetes.io/master=”

0
votes

Upgrade Kube-aws v1.15.5 cluster to the next version 1.16.8.

Use Case:

I want to keep the Same node label for Master and Worker nodes as I'm using in v1.15 .

When I tried to upgrade the cluster to V1.16 the --node-labels is restricted to use 'node-role'

If I keep the node role as "node-role.kubernetes.io/master" the kubelet fails to start after upgrade. if I remove the label, kubectl get node output shows none for the upgraded node.

How do I reproduce?

Before the upgrade I took a backup of 'cp /etc/sysconfig/kubelet /etc/sysconfig/kubelet-bkup' have removed "-role" from it and once the upgrade is completed, I have moved the kubelet sysconfig by replacing the edited file 'mv /etc/sysconfig/kubelet-bkup /etc/sysconfig/kubelet'. Now I could able to see the Noderole as Master/Worker even after kubelet service restart.

The Problem I'm facing now?

Though I perform the upgrade on the existing cluster successfully. The cluster is running in AWS as Kube-aws model. So, the ASG would spin up a new node whenever Cluster-Autoscaler triggers it.

But, the new node fails to join to the cluster since the node label "node-role.kubernetes.io/master" exists in the code base.

How can I add the node-role dynamically in the ASG scale-in process?. Any solution would be appreciated.

Note: (Kubeadm, kubelet, kubectl )- v1.16.8

1
kuberneteskube-aws
Hi, can You check if the nodes created by autoscaler have matching kubeadm and kubelet version? Maybe You need to reconfigure node pool template for autoscaler after upgrading to newer version. - Piotr Malec
Yes all the binaries are latest. - JafferK8S

1 Answers

1
votes

I have sorted out the issue. I have created a Python code that watches the node events. So whenever ASG spins up a new node, after it joins to the cluster, the node wil be having a role "" , later the python code will add a appropriate label to the node dynamically.

Also, I have created a docker image with the base of python script I created for node-label and it will run as a pod. The pod will be deployed into the cluster and it does the job of labelling the new nodes.

Ref my solution given in GitHub https://github.com/kubernetes/kubernetes/issues/91664

I have created as a docker image and it is publicly available https://hub.docker.com/r/shaikjaffer/node-watcher

Thanks, Jaffer