Monitoring Kubernetes cluster using prometheus outside the k8 cluster

Question

• We have kubernetes cluster where I have service account "kube", namespace "monitoring" with cluster role binding created to monitor cluster
• We have prometheus installed on a linux system (on prem) outside the cluster and is installed using "root"
• When I try to connect to the k8 cluster with the https api using ca.crt and user token (given by kubernetes admin), it throws multiple errors.

Error messages:

component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig

component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig

Prometheus configuration:

  - job_name: 'kubernetes-apiservers'
    scheme: https
    tls_config:
      ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/prometheus-2.12.0.linux-amd64/ca.crt
    relabel_configs:
    - source_labels: [monitoring, monitoring-sa, 6443]
      action: keep
      regex: default;kubernetes;https

  - job_name: 'kubernetes-nodes'
    scheme: https
    tls_config:
        ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token

    kubernetes_sd_configs:
    - role: node
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/ca.crt
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: https://example.com:1234
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics

Answer 1

The main problem you're facing is: "unable to use specified CA cert /root/prometheus/ca.crt"

Someone recently faced the same problem: https://github.com/prometheus/prometheus/issues/6015#issuecomment-532058465

He solved it by reinstalling the new version.

Version 2.13.1 is out. Try installing the latest version, it might solve your problem too.

Answer 2

Your ca.crt is most probably still in base64 format since secrets are encoded that way when describing them, as explained here.

Answer 3

Maybe your ca.crt have some error, check your ca cert file, make sure this file format like this:

-----BEGIN CERTIFICATE-----
xxxxx
-----END CERTIFICATE-----

I think your ca.crt is get by kubectl get serviceaccount -o yaml, but this is a public key with your kubernetes cluster, so, if you want to get the token, you can specify the serviceAccountName in the yaml file with a new Deployment, like this:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: test
        version: v1
    spec:
      serviceAccountName: prometheus
      containers:
      - name: test
        image: alpine
        imagePullPolicy: Always
        command: ["ping", "127.0.0.1"]
      imagePullSecrets:
        - name: harbor-secret
      restartPolicy: Always

Then, get your token and ca.crt under /var/run/secrets/kubernetes.io/serviceaccount/.