Pod can't access kubelet endpoints

Question

When ever my pods try to query any kubelet endpoints (any port), they immediately get a connection refused.

My goal is to have prometheus get those metrics from cadvisor, and I am trying to create a ServiceMonitor to scrape the cadvisor metrics (for info on pod level resource usage).

Trying wget from any pod on my cluster results in the following:

$ wget 10.116.0.10:4194
Connecting to 10.116.0.10:4194 (10.116.0.10:4194)
wget: can't connect to remote host (10.116.0.10): Connection refused

I am running on DigitalOcean managed Kubernetes

edit: the above happens on any port, not just 4194

ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet
  labels:
    app: prometheus
spec:
  jobLabel: k8s-app
  endpoints:
  - port: cadvisor
    interval: 15s
    honorLabels: true
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      k8s-app: kubelet
      app.kubernetes.io/managed-by: prometheus-operator
      app.kubernetes.io/name: kubelet
  namespaceSelector:
    matchNames:
    - kube-system

On Kubernetes, cAdvisor is inside kubelet binary, as you should know. So you can install metrics-server and just scrap the data directly from kubernetes API as Prometheus was intended to do. — Hector Vido
@HectorVido I would love to scrape metrics-server, but it is not in Prometheus format, nor does it list all of the metrics on a single endpoint. — danthegoodman
When using kube-prometheus-stack, scraping kubelet cadvisor metrics should be enabled by default. If you are not using promethus stack you can still use kubelet/servicemonitor.yaml and see how its done. Is it any helpful information? — Matt
@Matt I've been trying to decipher the helm file. I'm not using helm in the install so I need to convert that to a normal ServiceMonitor but I will try! — danthegoodman
@Matt Something I had done following that must have been it, maybe a different port or using the right TLS config or both but it works now! — danthegoodman

Paul Rey Paul Rey · Accepted Answer · 2021-03-19T10:38:59

The error "Connection refused" indicates a firewall could block the port or no service is listening on the port. Are you sure using the right port?

The port 4194 is the port used by cAdvisor, why would you like to access it with an IP (10.116.0.10)? It has nothing related to Kubelet.

Pod can't access kubelet endpoints

1 Answers