Scrape metrics of kubernetes containers with prometheus for HTTP and HTTPS ports

Question

We want our Prometheus installation to scrape the metrics of both containers within a pod. One container exposes the metrics via HTTPS at port 443, whereas the other container exposes them via HTTP at port 8080. Both containers provide the metrics at the same path, namely /metrics.

If we declare the prometheus.io/scheme to be either http or https, only one container will be scraped. For the other one we always receive: server returned HTTP status 400 Bad Request The same happens if we do not define the prometheus.io/scheme at all. Prometheus will then use http for both ports, and fail for the container that exposes the metrics at port 443 as it would expect HTTPS requests only.

Is there a way to tell prometheus how exactly it shall scrape the individual containers within our deployment? What are feasible workarounds to acquire the metrics of both containers?

Versions

Kubernetes: 1.10.2

Prometheus: 2.2.1

Deployment excerpt

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xxx
  namespace: xxx
spec:
  selector:
    matchLabels:
      app: xxx
  template:
    metadata:
      labels:
        app: xxx
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: container-1
        image: xxx
        ports:
        - containerPort: 443
      - name: container-2
        image: xxx
        ports:
        - containerPort: 8080

Prometheus configuration:

- job_name: kubernetes-pods
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names: []
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    separator: ;
    regex: ([^:]+)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_pod_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: kubernetes_namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_pod_name
    replacement: $1
    action: replace

Have you considered using a sidecar container (such as haproxy) to smooth over the protocol imbalance between the two current containers? I think that you can name the sidecar container container-1, change the current container-1 to something else, and then (from Prometheus's PoV) the metrics will appear with their correct name, and only you would know about the trickery. I didn't see anything in the discovery source that would allow the fine-grained control you're describing — mdaniel
Hi Matthew. We could do that, and in fact for most of our scenarios one of the two containers already is a sidecar to offload TLS traffic. More or less, we already have a workaround in place doing exactly this and scraping all metrics from 443. Still we have other constellations where we would love to prevent deploying just another sidecar — croeck

ahus1 ahus1 · Accepted Answer · 2018-05-06T19:13:09

I found a GIST snippet that takes the port from the container directly if it is named "metrics", instead of relying on a per-pod annotation. It also contains a comments to make this a regex for any port that starts with "metrics".

Maybe you can extend it to also extract the schema from the port name, like "metrics-http" and "metrics-https".

https://gist.github.com/bakins/5bf7d4e719f36c1c555d81134d8887eb

# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this. This
#    will be the same for every container in the pod that is scraped.
# * this will scrape every container in a pod with `prometheus.io/scrape` set to true and the
    port is name `metrics` in the container
# * note `prometheus.io/port` is no longer honored. You must name the port(s) to scrape `metrics`
#   Also, in some of the issues I read, there was mention of a container role, but I couldn't get 
#   that to work - or find any more info on it.
- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: metrics
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [ __address__, __meta_kubernetes_pod_container_port_number]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

Scrape metrics of kubernetes containers with prometheus for HTTP and HTTPS ports

Versions

Deployment excerpt

Prometheus configuration:

2 Answers