Configuring a jmx-prometheus-exporter docker container to read jmx local application's metrics

Question

I am trying to visualise my Spring application (still on localhost until now) metrics with grafana. To do that, first I configured a connection pool with HikariCP and I added monituring features with Jmx (MBean). Until this step every things works fine.

Then I started Grafana, a Prometheus and a jmx-prometheus-exporter docker containers.

Here is my docker compose file

version: '3'

services:

  grafana:
    image: grafana/grafana:latest
    ports:
      - 3000:3000

  prometheus:
    image: prom/prometheus:latest
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  jmx-exporter:
    image: sscaling/jmx-prometheus-exporter
    ports:
      - 5556:5556

And here is prometheus configuration file

scrape_configs:
  - job_name: 'prometheus'

    static_configs:
      - targets: ['localhost:9090']
        labels:
          group: 'prometheus'

  - job_name: 'app-server'

    static_configs:
      - targets: ['localhost:8080']


  - job_name: 'jmx-exporter'

    static_configs:
      - targets: ['jmx-exporter:5556']

in the prometheus: "http://localhost:9090/targets" I could see the jmx-prometheus-exporter and prometheus itself as healthy services.

When visiting http://localhost:5556/ I see some values, but there is nothing that shows that they are specific to my application ( I don't see the pool name I have set ...) and it seems logic because I don't see where how the jmx container is configured to read metrics from my application (local jmx)

# TYPE java_lang_MemoryPool_UsageThresholdSupported untyped
java_lang_MemoryPool_UsageThresholdSupported{name="Metaspace",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="PS Old Gen",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="PS Eden Space",} 0.0
java_lang_MemoryPool_UsageThresholdSupported{name="Code Cache",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="Compressed Class Space",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="PS Survivor Space",} 0.0
# HELP java_lang_Threading_ThreadContentionMonitoringEnabled ThreadContentionMonitoringEnabled (java.lang<type=Threading><>ThreadContentionMonitoringEnabled)
# TYPE java_lang_Threading_ThreadContentionMonitoringEnabled untyped
java_lang_Threading_ThreadContentionMonitoringEnabled 0.0
# HELP java_lang_OperatingSystem_CommittedVirtualMemorySize CommittedVirtualMemorySize (java.lang<type=OperatingSystem><>CommittedVirtualMemorySize)

My question is, I want to expose my application metrics in order to be read in the jmx-prometheus-exporter container ( https://github.com/sscaling/docker-jmx-prometheus-exporter), and I don't see how to configure that, ( the exporter java agent solution may work but I want something dockerized)

Answer 1

I was working on something similar. Your prometheus configuration file seems to be correct.

The only way I found was to use javaagent. You can download the javaagent jar from https://github.com/prometheus/jmx_exporter.

Then add this line to your java application :

javaagent:/YOUR_PATH/jmx_prometheus_javaagent-0.3.0.jar=PORT:/ANOTHER_PATH/config-jmx-tomcat.yaml"

I think that if you can't see your data the config for your jmx exporter isn't set right. Check this github for the syntax : https://github.com/pentachore/jmx_exporter_hikaricp

Answer 2

you have to add the config file for jmx exporter where you will be mentioning the jmx url to consume the metrics from and map the config file to override the file present inside the jmx exporter

eg: compose file

version: '2'
services:
  jmx-exporter:
    image: sscaling/jmx-prometheus-exporter
    ports:
      - "9072:9072"
    environment:
      SERVICE_PORT: 9072
    volumes:
      - ./config.yml:/opt/jmx_exporter/config.yml

here /opt/jmx_exporter/config.yml is the default file present inside the image and ./config.yml is the file where you have to mention the configuration. once mapped the config file you defined will override the default image file.

eg : config file

startDelaySeconds: 0
lowercaseOutputName: false
lowercaseOutputLabelNames: false
ssl: false
jmxUrl: service:jmx:rmi:///jndi/rmi://[jmxhost:port]/jmxrmi

[jmxhost:port] - you have to substitute this with the actual name of the jmx host and jmx port. There are other configuration that can be added to config.yml based on your requirement.

Answer 3

You need to provide a config file for jmx exporter: refer link

For example, if your application is running on port 8080 then provide config like

hostPort: 127.0.0.1:8080

in config.yaml for jmx exporter.