Scraping custom stats from multiple Kubernetes pods via Prometheus

Question

We are running Prometheus on our cloud under Kubernetes, and are able to get stats back for the memory, CPU usage etc of the nodes. Now we want to be able to scrape our own custom time series from pods running on the cloud. As I understand it, this requires a job with a kubernetes_sd_config of role 'pod' (and a relabel that will select only pods with a given name). So far so good. However, from the docs I read:

The pod role discovers all pods and exposes their containers as targets. For each declared port of a container, a single target is generated. If a container has no specified ports, a port-free target per container is created for manually adding a port via relabeling.

So each pod must declare a port (in its k8s yaml, presumably?) otherwise it comes up port-free. In which case how would I ensure that each pod has a unique port number? (and if I didn't, it would presumably fail if two pods were started on the same node)

mdaniel mdaniel · Accepted Answer · 2017-12-02T05:41:23

and a relabel that will select only pods with a given name

I suppose that's possible, but in our config, we just omit the scrape annotation (discussed below) and Prom doesn't contact those Pods; we haven't touched the Prom config itself in months, but change the Pod annotations more frequently.

So each pod must declare a port

In their example config one can see where the annotations from the Pod are extracted and used to point the Prom scraper at the port and path for that specific Pod (including, of course, whether to scrape it at all).

metadata:
    annotation:
        prometheus.io/scrape: true
        prometheus.io/path: /custom-prom-metrics
        prometheus.io/port: 9145

That snippet could go into your ReplicationController, Deployment, ReplicaSet, Pod (if you deploy "bare" Pods), or attached to a Pod post-facto using the Kubernetes API. To the best of my knowledge, one can apply and remove those annotations at will, and Prom will pick it up in the next scrape.

how would I ensure that each pod has a unique port number? (and if I didn't, it would presumably fail if two pods were started on the same node)

Assuming you have a reasonable Kubernetes install, with a software defined network, then every Pod has its own IP address, and thus its own port namespace. If you wanted to standardize on every Pod using 9145, that would be perfectly fine.

Scraping custom stats from multiple Kubernetes pods via Prometheus

1 Answers