Prometheus Monitoring Issue for MySQL

Question

I'm trying to setup a monitoring for MySQL as mentioned in this Percona link

I'm setting this up for the first time.

This is my Prometheus config file:

    global:
  scrape_interval:     5s
  evaluation_interval: 5s
scrape_configs:
- job_name: linux
  static_configs:
  - targets:
    - '172.19.36.189:3306'
    labels:
      alias: db1

Prometheus version:

prometheus, version 1.1.2 (branch: master, revision: 36fbdcc)
build user: root@a74d279
build date: 20160908-13:12:43
go version: go1.6.3

While I check in Prometheus targets page: I get the following error:

There are no errors reported in Prometheus logs.

When I click the metric link, the metrics page is not opening. And the state of the target is DOWN.

I have started mysqld and node exporters properly as well.

Where is the issue?

Answer 1

You need to scrape the mysqld exporter (usually port 9104), not mysqld itself.

Answer 2

Three things to check when scrape targets are not reachable:

1. Networking:

   Is the scrape target http://172.19.36.189:9104/metrics reachable from where you've opened the Prometheus GUI in the browser? Check curl -vvv http://172.19.36.189:9104/metrics and any proxy to http://172.19.36.189:9104/metrics.

2. Prometheus logs:

   Start prometheus with debugging turned on by using the flag --log.level:

   $ /bin/prometheus -h
   ...
   --log.level=info      [debug,
                         info,
                         warn,
                         error]
   

   Then, check the container's logs using:

   docker logs <name of prometheus container>
   kubectl logs <name of prometheus pod>
   

3. Is the scrape target's port 9104 exposed in the container/pod/service?