Start/Stop local dev kubernetes cluster created by kubeadm (like microk8s or minikube)

2
votes

3 nodes Kubernetes cluster created using kubeadm v1.19.9. The nodes are VMs (using KVM hypervisor on Ubuntu 20.04).

The usage of this Kubernetes cluster is for development and exercises on Kubernetes. I'd like to stop and restart the cluster where it was left off. In the same fashion as the stop and start commands available with minikube or microk8s.

EDIT: clarify the question to avoid suggested duplicated posts. I am looking for an elegant solution to stop and restart the same cluster. NOT to destroy / reset / uninstall the cluster.

I couldn't find a simple solution from various web searches. There are solutions which suggested to tear down the cluster which is not my use case here. An answer dating 3 years ago, proper shutdown of a kubernetes cluster, is closer to what I want but this sounds quite complicate. Another solution How to Setup & Recover a Self-hosted Kubeadm Kubernetes Cluster After Reboot doesn't explain well enough about the principle used.

I hope there is a simpler solution now.

EDIT (2021-04-11): Kubernetes 1.21 release notes:

Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default.

kubernetes/enhancements Graceful node shutdown #2000

Enhancement target (which target equals to which milestone):

  • Alpha release target (1.20)
  • Beta release target (1.21)
  • Stable release target (1.23)
1
kuberneteskubeadm
You should be able to stop worker VMs, then master. When starting , start master first then workers. Your pods should spin up and work fine after after all kube-system pods are up(coreDNS etc). Have you tried that? – Arfat Binkileb
You might want to do that in a test environment. Just take back up of your Kubernets (following k8s docs). Logically it should work. – Arfat Binkileb
Of course this is a test env (used for local experimentation and practice). Shutting down the VM would be straightforward but I am not sure if this is OK. Maybe there is a proper way to gracefully terminate ? – Polymerase
"Maybe there is a proper way to gracefully terminate" - not really, but k8s should be able to easly handle shutdowns. What may not be able to handle it are the applications/containers that you run. I had a chance to see a cluster where applications had to be deployed in order, otherwise they wouldn't start (don't do this). ansible deployed containers that where then provisioned by the script. Make sure containers start on their own and don't require manual intervention and you should be fine. – Matt
The How to Setup & Recover a Self-hosted Kubeadm Kubernetes Cluster After Reboot mentions some scripts (additional setup) but this is due to SelfHosting. – Matt

1 Answers

0
votes

To summarize:

k8s should be able to handle shutdowns. What may not be able to handle it are the applications/containers that you run - just make sure containers start on their own and don't require manual intervention and you should be fine.

I mentioned in comments about flushing etcd data to disk but (after some research) this should not be neccessary since etcd does it itself and implements strong consistency model to make sure it doesn't lose the data. But this doesn't mean you should not be doing your backups - it's better to have a backup and don't ever use it then don't have one when needed.

The solution mentioned in How to Setup & Recover a Self-hosted Kubeadm Kubernetes Cluster After Reboot is relevant only if you use SelfHosting.

Also (for convenience) make sure that all configs are persisting between reboots e.g. swap partition should be disabled and if you only run swapoff -a it won't persist after reboot - it's much better to make changes in fstab so that when rebooted you don't have to disable anything again manually.

Here are some links:

Backing up an etcd cluster

etcd disaster recovery

Permanently Disable Swap for Kubernetes Cluster