Is there an ability to deploy Prometheus Operator on a distributed environment?

Question

What I have: Prometheus Operator Helm chart deployed on a single Kubernetes cluster the same which is also application cluster. So, all of the application-related pods, prometheus exporter pods, Grafana pod and Prometheus itself live within the same cluster.

What I want: I want to split the above configuration is such a way, that I have all the application-related as well as exporter pods exist in application cluster, while having Prometheus and Grafana pods deployed to separate one(in order to not have single point of failure)

Is there a way to achieve this using Prometheus Operator?

Answer 1

It's not quite what you're asking, but you could run these workloads in the same cluster, just on different nodes using node selectors or taints.

It would keep the monitoring/observability workloads separate from the application workloads without having to re-architect the cluster's network or deal with federation.

Answer 2

Technically, maybe, kind of. Using kubefed you can sync object state between clusters. You would need to have a mutually flat network so that the Prom pod in one cluster can talk directly to the target pods in the other.

Overall this is probably not a good idea though. Set up prom in each application cluster and then use prom-to-prom federation instead to have a single overall install.