Restarting master and losting pods on nodes

2
votes

I setup my cluster with one master and two nodes. I can create pods on nodes. If my master node fails (reboot) when I use kubeadm reset and then kubeadm init I lost all my pods, deployments, services.

Am I losting my pods because reset? What should I do?

Some similar questions:

https://stackpointcloud.com/community/question/how-do-i-restart-my-kubernetes-cluster

Is there a best practice to reboot a cluster

1
kuberneteskubeadm
Did you found a best practice to reboot your k8s cluster? I have the same problem, my k8s cluster on an vserver is completly crashed (all pods/containers are exited because of OOM after 4 Months running). My biggest concern is, to delete the whole configuration. From the answer of @janos lenart i would try to restart my OS system first. Any other suggestions? I'm using K8s 1.16 & Istio 1.3.0. - snukone
Restarting the vserver doesnt solved the problem. The api-server etc wasnt started. Its still all dead. - snukone
Got the solution: Setting fail-swap-on to false: vi /etc/sysconfig/kubelet (on CentOS 7) and insert following line: KUBELET_EXTRA_ARGS=--fail-swap-on=false. systemd starts kubelet automatically or you can do it manually (see here: github.com/kubernetes/kubernetes/issues/… ) - snukone

1 Answers

2
votes

kubeadm reset on the master deletes all configuration (files and a database too). There is no way back.

You should not run kubeadm init when you reboot the master. kubeadm init is a one off action to bootstrap the cluster. When the master is rebooted your OS's init system (systemd, upstart, ...) should start kubelet which in turn starts the master components (as containers). An exception is if your cluster is self-hosting