Metric server not working : unable to handle the request (get nodes.metrics.k8s.io)

8
votes

I am running command kubectl top nodes and getting error : 

node@kubemaster:~/Desktop/metric$ kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

Metric Server pod is running with following params : 

    command:
    - /metrics-server
    - --metric-resolution=30s
    - --requestheader-allowed-names=aggregator
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP

Most of the answer I am getting is the above params, Still getting error

E0601 18:33:22.012798       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:kubemaster: unable to fetch metrics from Kubelet kubemaster (192.168.56.30): Get https://192.168.56.30:10250/stats/summary?only_cpu_and_memory=true: context deadline exceeded, unable to fully scrape metrics from source kubelet_summary:kubenode1: unable to fetch metrics from Kubelet kubenode1 (192.168.56.31): Get https://192.168.56.31:10250/stats/summary?only_cpu_and_memory=true: dial tcp 192.168.56.31:10250: i/o timeout]

I have deployed metric server using : 

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

What am I missing? Using Calico for Pod Networking

On github page of metric server under FAQ: 

[Calico] Check whether the value of CALICO_IPV4POOL_CIDR in the calico.yaml conflicts with the local physical network segment. The default: 192.168.0.0/16.

Could this be the reason. Can someone explains this to me.

I have setup Calico using : kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml

My Node Ips are : 192.168.56.30 / 192.168.56.31 / 192.168.56.32

I have initiated the cluster with --pod-network-cidr=20.96.0.0/12. So my pods Ip are 20.96.205.192 and so on.

Also getting this in apiserver logs

E0601 19:29:59.362627       1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.100.152.145:443/apis/metrics.k8s.io/v1beta1: Get https://10.100.152.145:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

where 10.100.152.145 is IP of service/metrics-server(ClusterIP)

Surprisingly it works on another cluster with Node Ip in 172.16.0.0 range. Rest everything is same. Setup using kudeadm, Calico, same pod cidr

2
kuberneteskubernetes-metrics
can you just leave - --kubelet-insecure-tls and - --kubelet-preferred-address-types=InternalIP? – suren
Tried but not working, – Ankit Bansal
Idk then mate. I have deployed metrics server 100 times, and always started with this issue. You can set - --v=4 for more verbose output from the logs. – suren
Addresses within subnet 20.96.0.0/12 are public addresses so maybe this is why it is failing. – Matt

2 Answers

5
votes

It started working after I edited the metrics-server deployment yaml config to include a DNS policy.

hostNetwork: true

Click here to view the image description

Refer to the link below: https://www.linuxsysadmins.com/service-unavailable-kubernetes-metrics/

1
votes

Default value of Calico net is 192.168.0.0/16 There is a comment in yaml file:

The default IPv4 pool to create on startup if none exists. Pod IPs will be chosen from this range. Changing this value after installation will have no effect. This should fall within --cluster-cidr.

  • name: CALICO_IPV4POOL_CIDR value: "192.168.0.0/16"

So, its better use different one if your home network is contained in 192.168.0.0/16.

Also, if you used kubeadm you can check your cidr in k8s:

kubeadm config view | grep Subnet

Or you can use kubectl:

kubectl  --namespace kube-system get configmap kubeadm-config -o yaml

Default one in kubernetes "selfhosted" is 10.96.0.0/12