Prometheus in kubernetes have too many unhealthy targets

Question

First, My kubernetes cluster is based on Bare-metal environment.

cluster info:

k get no -o wide

NAME       STATUS   ROLES    AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
k-master   Ready    master   142d   v1.18.10   192.168.6.211   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://18.9.9
k-node-1   Ready    <none>   142d   v1.18.10   192.168.6.212   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://18.9.9
k-node-2   Ready    <none>   142d   v1.18.10   192.168.6.213   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://18.9.9

When i install Prometheus in cluster to monitor k8s, My step:

1. Create RBAC

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitor

2. Create ConfigMaps

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-conf
  namespace: monitor
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s #

    scrape_configs:

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    - job_name: 'kubernetes-service'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: service

    - job_name: 'kubernetes-endpoints'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: endpoints

    - job_name: 'kubernetes-ingress'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: ingress

    - job_name: 'kubernetes-pods'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: pod

3. Create Prometheus Server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: monitor
spec:
  selector:
    matchLabels:
      app: prometheus-server
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
      - name: prometheus-server
        image: prom/prometheus
        volumeMounts:
        - name: conf
          mountPath: "/etc/prometheus"
          readOnly: true
        ports:
        - containerPort: 9090
      serviceAccountName: prometheus
      volumes:
      - name: conf
        configMap:
          name: prometheus-conf

Anyway, service and ingress of prometheus related has already created, and i can access prometheus webUI, but so many Unhealthy Targets in Service Discovery page.

For kubernetes-nodes details: server returned HTTP status 403 Forbidden

I don't know how to fix it and other more. Can anybody teach me? thanks!

Answer 1

You can check the server side certification file for kubelet:

# openssl x509 -text -noout -in /var/lib/kubelet/pki/kubelet.crt

Maybe another file name. and check the SAN part of the output, like this:

 X509v3 Subject Alternative Name: 
                DNS:host.yourdomain.com 

If there is only the node name get displayed, then you access it through ip will not succeed. In this way, either probe the node through api-server, or regenerate the certification files with ip in the SAN field.