I have set up a v1.13 Kubernetes cluster using Kube spray. Our etcd is running as docker containers outside the K8s cluster. If I check the etcd certificates, I can see each etcd has its own ca, client cert and key.
If I want to scrape the /metrics endpoints of these etcd conatiners for Prometheus, which certificates to use for the HTTPS endpoints?
I am not yet sure, if this is the most secured way or not. But I took the ca.pem, cert and key that one of the etcd uses.
I created a Kubernetes secret object out of the three:
kubectl create secret generic etcd-metrics -n monitoring --from-file=etcd-secrets/
Then I added the secrets as configmaps in Prometheus config and below as my scrape
targets:
- job_name: etcd
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
static_configs:
- targets:
- 172.xxxxx:2379
- 172.xxxxx:2379
- 172.xxxxx:2379
tls_config:
ca_file: /etc/ssl/etcd/ca.pem
cert_file: /etc/ssl/etcd/etcd-node.pem
key_file: /etc/ssl/etcd/etcd-key.pem
insecure_skip_verify: false
While not exactly what you asked, I had great success pushing that authentication down onto the actual machine by using socat running in a sidecar container listening on etcd's prometheus port :9379 and then you can just point prometheus at http://${etcd_hostname}:9379/metrics without having to deal with authentication for those metrics endpoints.
I don't have the socat invocation in front of me, but something like:
socat tc4-listen:9379,reuseaddr,fork \
openssl:127.0.0.1:2379,capath=/etc/kubernetes/pki/etcd/cacert.crt,key=/etc/kubernetes/pki/etcd/peer.key,cert=/etc/kubernetes/pki/etcd/peer.crt
