Prometheus for k8s multi clusters

1
votes

I have 3 kubernetes clusters (prod, test, monitoring). Iam new to prometheus so i have tested it by installing it in my test environment with the helm chart:

# https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack

But if i want to have metrics from the prod and test clusters, i have to repeat the same installation of the helm and each "kube-prometheus-stack" would be standalone in its own cluster. It is not ideal at all. Iam trying to find a way to have a single prometheus/grafana which would federate/agregate the metrics from each cluster's prometheus server.

I found this link, saying about prometheus federation:

https://prometheus.io/docs/prometheus/latest/federation/

If install the helm chart "kube-prometheus-stack" and get rid of grafana on the 2 other cluster, how can i make the 3rd "kube-prometheus-stack", on the 3rd cluster, scrapes metrics from the 2 other ones?
thanks

2
kubernetesprometheus

2 Answers

3
votes

You have to modify configuration for prometheus federate so it can scrape metrics from other clusters as described in documentation:

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

    static_configs:
      - targets:
        - 'source-prometheus-1:9090'
        - 'source-prometheus-2:9090'
        - 'source-prometheus-3:9090'

params field checks for jobs to scrape metrics from. In this particular example

It will scrape any series with the label job="prometheus" or a metric name starting with job: from the Prometheus servers at source-prometheus-{1,2,3}:9090

You can check following articles to give you more insight of prometheus federation:

  1. Monitoring Kubernetes with Prometheus - outside the cluster!

  2. Prometheus federation in Kubernetes

  3. Monitoring multiple federated clusters with Prometheus - the secure way

  4. Monitoring a Multi-Cluster Environment Using Prometheus Federation and Grafana

0
votes

You could try looking at Wavefront. It's a commercial tool now but you can get a 30 day trial free - also, it understands promQL. So essentially, you could use the same prometheus rules and config across all clusters, and then use wavefront to just connect to all of those prom instances.

Another option may be Thanos, but I've never used it personally.