Coredns in pending state in Kubernetes cluster

6
votes

I am trying to configure a 2 node Kubernetes cluster. First I am trying to configure the master node of the cluster on a CentOS VM. I have initialized the cluster using 'kubeadm init --apiserver-advertise-address=172.16.100.6 --pod-network-cidr=10.244.0.0/16' and deployed the flannel network to the cluster. But when I do 'kubectl get nodes', I get the following output ----

[root@kubernetus ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
kubernetus   NotReady   master   57m   v1.12.0

Following is the output of 'kubectl get pods --all-namespaces -o wide ' ----

[root@kubernetus ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE
kube-system   coredns-576cbf47c7-9x59x             0/1     Pending   0          58m   <none>         <none>       <none>
kube-system   coredns-576cbf47c7-l52wc             0/1     Pending   0          58m   <none>         <none>       <none>
kube-system   etcd-kubernetus                      1/1     Running   2          57m   172.16.100.6   kubernetus   <none>
kube-system   kube-apiserver-kubernetus            1/1     Running   2          57m   172.16.100.6   kubernetus   <none>
kube-system   kube-controller-manager-kubernetus   1/1     Running   1          57m   172.16.100.6   kubernetus   <none>
kube-system   kube-proxy-hr557                     1/1     Running   1          58m   172.16.100.6   kubernetus   <none>
kube-system   kube-scheduler-kubernetus            1/1     Running   1          57m   172.16.100.6   kubernetus   <none>

coredns is in a pending state for a very long time. I have removed docker and kubectl, kubeadm, kubelet a no of times & tried to recreate the cluster, but every time it shows the same output. Can anybody help me with this issue?

5
dockerkuberneteskubectlkubeadmkubelet
First step for any problem is to get the logs - use kubectl logs podname --namespace=kube-system and also check if there is something in the events with the kubectl get events --namespace=kube-system command. – Praveen Sripati
I have two coredns pods which are in pending state . Hence I tried ' kubectl logs coredns-576cbf47c7-9x59x --namespace=kube-system ' and 'kubectl logs coredns-576cbf47c7-l52wc --namespace=kube-system' but it did'nt show any output . – Aditya Datta
For the command 'kubectl get events --namespace=kube-system' , I got some warnings ---- 62m Warning FailedScheduling Pod 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. 88m Warning Unhealthy Pod Liveness probe failed: Get 172.16.100.6:6443/healthz: net/http: TLS handsh ake timeout. 88m Warning Unhealthy Pod Liveness probe failed: HTTP probe failed with statuscode: 403 – Aditya Datta
I also tried systemctl status kubelet -l and there were some errors --- Unable to update cni config: No networks found in /etc/cni/net.d ..... Oct 02 19:21:32 kubernetus kubelet[19007]: E1002 19:21:32.886170 19007 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized – Aditya Datta
Is that really kubernetus? – Rico

5 Answers

3
votes

Unable to update cni config: No networks found in /etc/cni/net.d ..... Oct 02 19:21:32 kubernetus kubelet[19007]: E1002 19:21:32.886170 19007 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

According to this error, you forgot to initialize a Kubernetes Pod network add-on. Looking at your settings, I suppose it should be Flannel.

Here is the instruction from the official Kubernetes documentation:

For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init.

Set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4 traffic to iptables’ chains. This is a requirement for some CNI plugins to work, for more information please see here.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

Note that flannel works on amd64, arm, arm64 and ppc64le, but until flannel v0.11.0 is released you need to use the following manifest that supports all the architectures:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml

For more information, you can visit this link.

3
votes

Try to install Pod network add-on (Base on this guide).

Run this line:

kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml
1
votes

For the Kubernetes cluster to be available, the cluster should have a Container Networking Interface (CNI). A pod-network is required to be configured for the dns pod to be functional.

Install any of the CNI Providers like: - Flannel - Calico - Canal - WeaveNet, etc.,

Without this, the hosted Kubernetes cluster would have the master in the NotReady State.

0
votes

Check if docker and kubernetes are using the same cgroup driver. I faced the same issue (CentOS 7, kubernetes v1.14.1), and setting same cgroup driver (systemd) fixed it.

0
votes

I installed kubernetes with 1 master + 1 work-node.
After I made kubeadm init ..., I faced two issues:

  1. On the master node, the coredns were pending.
  2. On the work-node, kubectl command didn't work out

On the work-node, I did the following and fixed the both issues:

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/kubelet.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config**