As the state of this cluster, how exactly it was deployed as well as the output messages of kubectl
, kubelet
are unknown, I will try to give some troubleshooting steps and circumstances that can lead to resolving some of the problems that are encountered here.
- Kubeadm
- Flannel
- Kubelet
- Reproduction
- Additional links
Kubeadm
The first thing is provisioning process of the Kubernetes cluster with kubeadm
:
Check all the requirements and steps with official documentation:
When invoking $ kubeadm init
command please make sure to add parameters like:
--apiserver-advertise-address=IP_ADDRESS_OF_MASTER_NODE
--pod-network-cidr=POD_NETWORK_CIDR_WITH_ACCORDANCE_TO_CNI
Provisioning cluster without --pod-network-cidr
parameter could lead to CNI related issues.
For Flannel default pod network cidr is 10.244.0.0/16
.
After kubeadm init
you need to apply one of many CNI's (like Flannel or Calico) as the kubeadm
tool does not do it automatically.
Please check if all of the nodes are in Ready
status with command:
$ kubectl get nodes -o wide
Output of this command should look like that:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k1 Ready master 69m v1.17.3 10.156.0.29 <none> Ubuntu 18.04.4 LTS 5.0.0-1031-gcp docker://19.3.7
k2 Ready <none> 65m v1.17.3 10.156.0.30 <none> Ubuntu 18.04.4 LTS 5.0.0-1031-gcp docker://19.3.7
k3 Ready <none> 63m v1.17.3 10.156.0.35 <none> Ubuntu 18.04.4 LTS 5.0.0-1031-gcp docker://19.3.7
Additionally you can $ kubectl describe node NAME_OF_THE_NODE
to get more detailed information about each of the nodes.
Flannel
There is official documentation about troubleshooting Flannel related issues: Github.com: flannel troubleshooting
The message that you received from kubelet
:
Warning FailedCreatePodSandBox 34m kubelet, slave-node Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "26cdaf3170806455a4731218d20c482bb2a41ded6ef85c90b560058e332df684" network for pod "label-demo": networkPlugin cni failed to set up pod "label-demo_default" network: open /run/flannel/subnet.env: no such file or directory
Is telling that there is subnet.env
file missing on a node that was supposed to schedule a sandbox
pod.
Furthermore please check if Flannel pods are running correctly. You can check it with either:
$ kubectl get pods -A
$ kubectl get pods -n kube-system
Additionally you can check logs of this pods by running command:
$ kubectl logs NAME_OF_FLANNEL_CONTAINER
Kubelet
You can check kubelet
logs by (on systemd os) with:
$ systemctl status kubelet
$ journalctl -u kubelet
Reproduction
I've managed to reproduce error that you encountered and it was happening when:
--pod-network-cidr=CIDR
was not used with $ kubeadm init
- Flannel CNI was applied after
$ kubeadm init
without --pod-network-cidr
Additional links:
There is an article talking about troubleshooting networking: Kubernetes.feisky.xyz: Network.
Please let me know if you have any other questions
kubeadm
or something likekubespray
? – Dawid Krukkubectl get nodes -o wide
andkubectl get pods -A
– Dawid Kruk