Querying external etcd with prometheus

Question

I've got prometheus running ontop of kubernetes with the following scrape config, as described by the documentation. Where the .pem files are located on disk within the prometheus container.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#

scrape_configs:
- job_name: etcd
  static_configs:
  - targets: ['10.0.0.222:2379','10.0.0.221:2379','10.0.0.220:2379']
  tls_config:
  # CA certificate to validate API server certificate with.
    ca_file: /prometheus/ca.pem
    cert_file: /prometheus/cert.pem
    key_file: /prometheus/key.pem

I see that etcd as a target in prometheus, however its returning garbage.

https://i.imgur.com/rdRI4V7.png

I am able to hit the metrics endpoint doing a local curl by passing in the client certificate information like so.

What am I doing wrong?

sudo curl --cacert /etc/ssl/etcd/ssl/ca.pem https://127.0.0.1:2379/metrics -L --cert /etc/ssl/etcd/ssl/node-kubemaster-rwva1-prod-2.pem --key /etc/ssl/etcd/ssl/node-kubemaster-rwva1-prod-2-key.pem^C

Answer 1

You need to add scheme: https for HTTPS scraping.