Prometheus with several servers

2
votes

I am starting to look at the Prometheus project and prototype it. So far I have implemented a few counters in my application (running on Weblogic), exposed the metrics servlet (dedicated war file), started on a dedicated machine a Prometheus server + Grafana using Docker images. Everything's working fine, the server scraps the counters and Grafana displays them in graphs.

But I am facing an issue because of my runtime environment. Basically I target my application via https://<myserver>/<myproduct>, and target the metrics via https://<myserver>/prometheus/metrics/, but then behind the scene I can hit two different Weblogic servers, that are not exposed outside of the platform they are deployed in (so my machine hosting the Prometheus server cannot reach the Weblogic instances directly). So I get different metrics depending on the server that will chosen to be hit by the load balancer.

Is there a way to workaround this kind of setup? A way to tell Prometheus server to aggregate the results somehow?

Thanks for your help!

3
prometheus
What product are you using in the role of load balancer? – b0tting
I believe this is done by an Nginx instance (not 100% sure as it's managed by a dedicated team) – Xendar

3 Answers

1
votes

Prometheus needs to talk to each instance directly, so going via any form of load balancer won't work. You should see if there's a way to have direct access.

0
votes

Most likely you should proxy each of them to a single instance on a different port, and then just set the list of targets with proper ports in place

0
votes

@Xendar I've configured to access with several hosts as follows,

static_configs:
- targets: ['192.168.x.x:9100']
- targets: ['192.168.x.y:9100']
- targets: ['192.168.x.z:8080']