Cannot capture client metrics with Prometheus

Question

I have a newbie question.

I'm using dockprom (github.com/stefanprodan/dockprom) to capture metrics from a docker-compose successfully.

Now I'm trying to monitor specific metrics from my applications using golang's Prometheus client library, but Prometheus shows my endpoint as down (0), with the message, in the targets section,

Get http://localhost:8090/metrics: dial tcp 127.0.0.1:8090: connect: connection refused

However, if I navigate to http://localhost:8090/metrics I can see the metrics being exposed.

Prometheus is running in a docker-compose set of containers, while my application is running in another.

The declaration of my endpoint in prometheus/prometheus.yml is:

job_name: 'cloud_server_auth'

scrape_interval: 10s

static_configs:

targets: ['localhost:8090']

I noticed that cAdvisor was failing when not running in privileged_mode, but even after fixing that, I still can't get prometheus to consume my metrics.

Any thoughts?

Thanks in advance to any who might shed some light on this issue, and please let me know if you need any further information. Adolfo

David Maze David Maze · Accepted Answer · 2020-03-14T04:43:16

If you're running Prometheus in a Docker container, then when Prometheus makes calls to other places to collect metrics, localhost is interpreted relative to the Prometheus container, which is to say, Prometheus is trying to collect metrics from itself.

If this is all running within the same docker-compose.yml file then you can use the Docker Compose services: name of the other container(s) as hostname(s) when configuring metric target(s). The target containers don't necessarily need to have published ports:, and you need to use the port number the process inside the container is running on – if your ports: remap a container port to a different host port, use the second (container) port number, not the first (host).

This is the same setup as other service-to-service calls within the same docker-compose.yml file. Networking in Compose has more details on the container network environment.

Cannot capture client metrics with Prometheus

1 Answers