Pod level monitoring in Prometheus for application deployed in different kubernetes cluster

Question

So I have 3 clusters of Kubernetes. In one of them, I have deployed Prometheus (and some other apps which I am monitoring). But I also want to monitor Applications (spring boot) that are in other clusters, at the pod level.

I have looked into different solutions like using the hierarchial-federation deployment for Prometheus but that really looks a bit messy. Many have suggested Service Discovery, but there are no resources for the same to monitor outside the cluster.

I am still a beginner in K8, so can you guys suggest to me how to do Pod level monitoring in Prometheus for applications that are deployed in a different Kubernetes cluster?

Shirine Shirine · Accepted Answer · 2021-03-04T05:33:42

You certainly cannot use service monitors and pod monitors as long as Prometheus does not reside in same cluster as your apps. If your applications already expose prometheus style metrics, you may want to take a look at

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md

Eg:

additionalScrapeConfigs: 
- job_name: <a job name>
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets: ["app1:9102", "app2:9102", "app3:9102"]

This solution is limited since it needs configuration at Prometheus side, so will not auto discover your metrics endpoints and scale.

Pod level monitoring in Prometheus for application deployed in different kubernetes cluster

1 Answers