Why does Kubernetes showing the nodes as ready even if they are not reachable?

Question

I am running Kubernetes cluster which is configured with a master and 3 nodes.

#kubectl get nodes
NAME       STATUS    AGE
minion-1   Ready     46d
minion-2   Ready     46d
minion-3   Ready     46d

I have launched couple of pods in the cluster and found that the pods are in pending state.

# kubectl get pods -o wide
NAME      READY     STATUS    RESTARTS   AGE       IP        NODE
httpd     0/1       Pending   0          10m       <none>
nginx     0/1       Pending   0          11m       <none>

One of the pod "httpd" yaml file:

# cat http.yaml
apiVersion: v1
kind: Pod
metadata:
  name: httpd
  labels:
    env: test
spec:
  containers:
  - name: httpd
    image: httpd

While debugging the reason for failure found that the couple of nodes configured are not ready. Only one node is reachable from master.

# ping minion-1
PING minion-1 (172.31.24.204) 56(84) bytes of data.
64 bytes from minion-1 (172.31.24.204): icmp_seq=1 ttl=64 time=0.575 ms

Whereas other nodes are not reachable:

# ping minion-2
PING minion-2 (172.31.29.95) 56(84) bytes of data.
From master (172.31.16.204) icmp_seq=1 Destination Host Unreachable

# ping minion-3
PING minion-3 (172.31.17.252) 56(84) bytes of data.
From master (172.31.16.204) icmp_seq=1 Destination Host Unreachable

The queries that I have here is

1) Why does Kubernetes showing the nodes as ready even if they are not reachable from master?

2) Why are the pods creation failing?

  Is it because of unavailability of nodes or any configuration issue in yaml file?

#  kubectl describe pod httpd
Name:           httpd
Namespace:      default
Node:           /
Labels:         env=test
Status:         Pending
IP:
Controllers:    <none>
Containers:
  httpd:
    Image:                      httpd
    Port:
    Volume Mounts:              <none>
    Environment Variables:      <none>
No volumes.
QoS Class:      BestEffort
Tolerations:    <none>
No events.

Following are the Kubernetes and etcd versions.

]# kubectl --version
Kubernetes v1.5.2
[root@raghavendar1 ~]# et
etcd        etcdctl     ether-wake  ethtool
[root@raghavendar1 ~]# etcd --version
etcd Version: 3.2.5
Git SHA: d0d1a87
Go Version: go1.8.3
Go OS/Arch: linux/amd64

3h4x 3h4x · Accepted Answer · 2017-10-15T18:47:34

Kubernetes do not use ICMP protocol to check nodes master node connectivity.

Nodes become Ready when the communication node -> api-server works and this is done via https protocol.

You can read more about about node - master connectivity in kubernetes documentation https://kubernetes.io/docs/concepts/architecture/master-node-communication/

Why pod isn't scheduled? The answer to this question is in the master logs probably, check kube-apiserver.log, kube-scheduler.log. The reason is cluster misconfiguration.

For start run it in a single network to get a grip of things and double check routing.

Why does Kubernetes showing the nodes as ready even if they are not reachable?

1 Answers