How to get TLS certs into pod for use with Prometheus helm chart?

Question

Here is the helm chart values for stable/prometheus: https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml

I was able to get this to work:

helm upgrade --install prometheus stable/prometheus \
--set extraScrapeConfigs="- job_name: 'myjob'
  scrape_interval: 1s
  metrics_path: /metrics
  scheme: https
  static_configs:
    - targets: ['###.##.###.###:#####']
  tls_config:
    ca_file: /prometheus/ca.pem
    key_file: /prometheus/key.pem
    cert_file: /prometheus/cert.pem
    insecure_skip_verify: true"

In order to do this I had to do:

kubectl cp localdir/ca.pem prometheus-server-abc:/prometheus -c prometheus-server
kubectl cp localdir/key.pem prometheus-server-abc:/prometheus -c prometheus-server
kubectl cp localdir/cert.pem prometheus-server-abc:/prometheus -c prometheus-server

I believe there's a better and more proper way to do this with Secret and mountPath. I tried something like the following with no luck:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
data:
  ca.pem: base64encodedcapem
  key.pem: base64encodedkeypem
  cert.pem: base64encodedcertpem

kubectl apply -f mysecret

helm upgrade --install prometheus stable/prometheus \
--set extraSecretMounts="- name: mysecret-mount
  mountPath: /somepathinpod/mysecret
  secretName: mysecret" \
--set extraScrapeConfigs="- job_name: 'myjob'
  scrape_interval: 1s
  metrics_path: /metrics
  scheme: https
  static_configs:
    - targets: ['###.##.###.###:#####']
  tls_config:
    ca_file: /somepathinpod/mysecret/ca.pem
    key_file: /somepathinpod/mysecret/key.pem
    cert_file: /somepathinpod/mysecret/cert.pem
    insecure_skip_verify: true"

I expected the certs to magically show up at /somepathinpod but they did not.

I'm assuming I don't have to clone the whole repo and manually edit the helm chart to put a volumeMount into the prometheus-server deployment/pod and can just change my helm command somehow. Any advice on how to get my certs in there?

Joe Joe · Accepted Answer · 2019-09-07T09:40:29

According to the documentation, the correct key to use would be server.extraSecretMounts instead of just extraSecretMounts.

Also verify the generated YAML on Kubernetes to contain the correct mounts via:

kubectl get deployment prometheus-server-object-name -o yaml

override.yaml

server:
  extraSecretMounts:
    - name: mysecret-mount
      mountPath: /etc/config/mysecret
      secretName: mysecret

extraScrapeConfigs: |
  - job_name: myjob
    scrape_interval: 15s
    metrics_path: /metrics
    scheme: https
    static_configs:
      - targets:
          - ###.##.###.###:#####
    tls_config:
      ca_file: /etc/config/mysecret/ca.pem
      key_file: /etc/config/mysecret/key.pem
      cert_file: /etc/config/mysecret/cert.pem
      insecure_skip_verify: true

helm upgrade -f override.yaml prometheus stable/prometheus

How to get TLS certs into pod for use with Prometheus helm chart?

1 Answers