Kube-state-metrics error: Failed to create client: ... i/o timeout

Question

I'm running Kubernetes in virtual machines and going through the basic tutorials, currently Add logging and metrics to the PHP / Redis Guestbook example. I'm trying to install kube-state-metrics:

git clone https://github.com/kubernetes/kube-state-metrics.git kube-state-metrics
kubectl create -f kube-state-metrics/kubernetes

but it fails.

kubectl describe pod --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7

...

Warning Unhealthy 28m (x8 over 30m) kubelet, kubernetes-node1 Readiness probe failed: Get http://192.168.129.102:8080/healthz: dial tcp 192.168.129.102:8080: connect: connection refused

kubectl logs --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7 -c kube-state-metrics

I0514 17:29:26.980707 1 main.go:85] Using default collectors
I0514 17:29:26.980774 1 main.go:93] Using all namespace
I0514 17:29:26.980780 1 main.go:129] metric white-blacklisting: blacklisting the following items:
W0514 17:29:26.980800 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0514 17:29:26.983504 1 main.go:169] Testing communication with server
F0514 17:29:56.984025 1 main.go:137] Failed to create client: ERROR communicating with apiserver: Get https://10.96.0.1:443/version?timeout=32s: dial tcp 10.96.0.1:443: i/o timeout

I'm unsure if this 10.96.0.1 IP is correct. My virtual machines are in a bridged network 10.10.10.0/24 and a host-only network 192.168.59.0/24. When initializing Kubernetes I used the argument --pod-network-cidr=192.168.0.0/16 so that's one more IP range that I'd expect. But 10.96.0.1 looks unfamiliar.

I'm new to Kubernetes, just doing the basic tutorials, so I don't know what to do now. How to fix it or investigate further?

EDIT - additonal info:

kubectl get nodes -o wide

NAME                STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kubernetes-master   Ready    master   15d   v1.14.1   10.10.10.11   <none>        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2
kubernetes-node1    Ready    <none>   15d   v1.14.1   10.10.10.5    <none>        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2
kubernetes-node2    Ready    <none>   15d   v1.14.1   10.10.10.98   <none>        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2

The command I used to initialize the cluster:

sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=192.168.0.0/16

please provide results of kubectl get nodes -o wide Are you using virtualbox, if yes is it bridged or host adapter mode? Did you use --apiserver-advertise-address= flag on kubeadm init? I will try to recreate as with 2 ubuntu machines and kubeadm I was able to run this and it works well. — aurelius
@aurelius I've added more details above, and to answer your questions: in Virtualbox I'm using both a bridged network (for internet access) and a host adapter network (because some tutorial said the nodes should be in such a network when experimenting in Virtualbox). I used --apiserver-advertise-address, more info in the post. — Rafał G.
I've also just tried creating a cluster with NAT networking instead of a bridge (with no changes in the host-only network except for a fresh subnet). The results were the same. — Rafał G.
Hi Rafał, if this solves the problem please accept the answer. If you have more questions related to the case feel free to ask. — aurelius
@aurelius Will do, I'm looking forward to trying your solution, I just need to find some time to try it out. — Rafał G.

aurelius aurelius · Accepted Answer · 2019-06-11T16:39:35

The reason for this is probably overlapping of Pod network with Node network - you set Pod network CIDR to 192.168.0.0/16 which your host-only network will be included into as its address is 192.168.59.0/24.

To solve this you can either change the pod network CIDR to 192.168.0.0/24 (it is not recommended as this will give you only 255 addresses for your pod networking)

You can also use different range for your Calico. If you want to do it on a running cluster here is an instruction.

Also other way I tried:

edit Calico manifest to different range (for example 10.0.0.0/8) - sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=10.0.0.0/8) and apply it after the init.

Another way would be using different CNI like Flannel (which uses 10.244.0.0/16).

You can find more information about ranges of CNI plugins here.

Kube-state-metrics error: Failed to create client: ... i/o timeout

1 Answers