How to configure Prometheus consul_sd_configs to use https for service

2
votes

We use Consul Registrator to register Docker containers with Consul as services.

Our prometheus is configured using consul_sd_configs to scrape a list of services resolved via Consul.

Recently a new Docker container was deployed which uses https with an internal certificate, rather than plain http. Prometheus is still trying to reach the service over http, which fails, and the service is then marked as down.

Is there any way to config Prometheus to use https for this service? Are we missing something in Consul or Registrator?

We cannot use a static config for the service, since the hosts and ports might change when it is deployed again - the service is spread over multiple Docker servers.

1
dockerprometheusconsul
were you able to get a solution for this problem? - Vijayakumar Udupa
I was able to resolve, posted solution here for next person. - Vijayakumar Udupa

1 Answers

2
votes

You can resolve this with help of relabel_configs & tls_config sections for consul_sd_configs settings in Prometheus configuration.

Below settings are using relabel_configs to set https scheme if service name is either SERVICE_NAME1 or SERVICE_NAME2, for any other service, scheme will have default value of http. You can use other labels also to identify list of services/instances where https is required.

  - job_name: 'consul_adapter_1'
    metrics_path: '/actuator/prometheus'

    #This handles Self Signed SSL Issue
    tls_config:
      insecure_skip_verify: true

    consul_sd_configs:
      - server: '<consul host>:<port>'
    relabel_configs:
      - source_labels: [__meta_consul_service]
        regex: (SERVICE_NAME1|SERVICE_NAME2)
        target_label: __scheme__
        replacement: https