How to Configure Prometheus running as container to scrape metrics generated by other application as container

Question

I am figuring out Network issues between Prometheus and python application within Docker. How can I make Prometheus able to scrape metrics generated by the python application within docker, and shows it on the Prometheus end.

Docker-compose.yml

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.0.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    links:
      - web
  web:
    image: python:3.5-alpine
    build: ./test
    ports:
     - "5000:8002"

Dockerfile:

FROM python:3.5-alpine
ADD app1.py /
RUN pip install prometheus_client
CMD ["python", "./app1.py"]
EXPOSE 8002

prometheus.yml

# my global config
global:
  scrape_interval:     15s 
  evaluation_interval: 15s 
  external_labels:
      monitor: 'codelab-monitor'
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['docker.for.mac.host.internal:9090']
  - job_name: 'python_app'
    metrics_path: /  
    static_configs:
      - targets: ['docker.for.mac.host.internal:8002']

So far, I can see Prometheus is running well, and its targets' status shows UP on 'docker.for.mac.host.internal:9090' and 'docker.for.mac.host.internal:8002'

And the python application is running as well, I can see the output metrics on the port

So everything should work by now, Prometheus can scrape metrics over the specified port. However there is no such metrics there.

you have created a link with name 'web', have you tried to use it in 'static_configs.targets'? something like 'web:8002'. IMO 'docker.for.mac.host.internal' still hits same container - the one with prometheus. — freakman

Joseph Larson Joseph Larson · Accepted Answer · 2019-02-07T17:05:54

This is the relative portion of a prometheus.yml file I'm using:

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 10s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    target_groups:
      - targets: ['localhost:9090']

  - job_name: app-server
    target_groups:
      - targets: ['dbserver:5000']

Note also you'll want to verify by hitting the various URLs yourself that there are metrics to pull.

curl "http://dbserver:5000/metrics"

Docker for Mac. The app running on dbserver:5000 is running on the Mac, not inside Docker. If it were inside Docker, I got that to work by making sure it was exposed and that I could hit it with curl from a Mac terminal window.

How to Configure Prometheus running as container to scrape metrics generated by other application as container

1 Answers