How to scale prometheus in kubernetes environment

Question

I have reached moment when I need to split my prometheus into smaller ones. I have been reading about it here but it does not say anything about scaling in kubernetes. Below is my setup:

• one node of prometheus
• one node of kube state metrics
• node exporter on each cluster node

and there are about 50 namespaces which produces thousands of metrics and one current setup with one prometheus is not enough. So I decided to split it to three instances like:

• one for kube state metrics metrics
• one for node exporter metrics
• one for kubernetes metrics

But after while i realised that those metrics are scraped by kubernetes_sd_config and there is no way to tell which metrics I want to scrape by which instance of prometheus or I am wrong. One solution would be to split kubernetes cluster into smaller one but it is too much work for now.

So my question is if there is any possibility to tell prometheus that I want scrape only kube state metrics, node exporter or native kubernetes metrics ?

Answer 1

Another option would be going for a horizontally scalable, distributed Prometheus implementation: https://github.com/weaveworks/cortex (NB I wrote this.)

Its not ready for prime time yet, but we're using it internally and getting pretty good results. It will be more effort to setup and operate than upstream Prometheus, but it should scale virtually indefinitely - and what's more we run it on Kubernetes, so it's really at home there.

Let me know if you're interested and I can walk you though setting it up.

Answer 2

Scaling in Kubernetes is the same as elsewhere. This is a question of using service discovery and relabelling to pick out what is monitored.

For example the configuration for the node exporters should already be a separate scrape_config so splitting it out to a separate Prometheus should be straightforward by splitting the configuration file.

Answer 3

I had a similar task for federation. Here is how I did, following @brian-brazil answer:

• setup a master prometheus with config:

scrape_configs: - job_name: dc_prometheus honor_labels: true metrics_path: /federate params: match[]: - '{job="my-kubernetes-job"}' static_configs: - targets: - prometheus-slaveA:9090 - prometheus-slaveB:9090 See how slaves are declared here. It's quite static for sure. Also, the match[] param here tells to grab all slave metrics. You would have to be smarter than that of course.

• setup slaves with that particular relabeling:

relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_slave] action: keep regex: slaveA

and similar for slaveB and so on.

Now for each pod, instead of having the well-known annotation prometheus.io/scrape: true|false, you would have prometheus.io/slave: slaveA|slaveB.

I described it with more details here: http://devlog.qaraywa.net/?p=176