spring boot actuator metrics monitoring with prometheus

Question

I have kubernetes cluster with multiple services, each service has an actuator and an health-check (readiness+liveness probes) I tried to create a Prometheus job for one of the services to scrape the response (200 response indicating service is good), But from my understanding, Prometheus is storing the data as a "time series" key-value so I'm not sure I am doing it right. This is the actuator response:

{
  "status" : "UP",
  "redisson" : {
    "status" : "UP"
  },
  "jedis" : {
    "status" : "UP",
  }
}

And this is my job:

  - job_name: 'service'
    scheme: https
    metrics_path: /health
    kubernetes_sd_configs:
      - role: service
        namespaces:
          names:
          - default
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        action: keep
        regex: service
      - target_label: __address__
        replacement: service.iot.test.service.co.il:8090
      - source_labels: [__param_target]
        target_label: instance                    
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: default
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service

Is there a way to scrape the json response that holds the "status"? And the "INVALID" is not a valid start token is caused by a bad scraping?

Oleg Mayko Oleg Mayko · Accepted Answer · 2019-04-03T17:39:09

Prometheus has a strict format definition. A quick solution would be just to change the endpoint to whatever is defined in springboot app (/actuator/prometheus or so). And check the metric "up" which is generated by prometheus for each target. I haven't tried Prometheus operator yet. so I cannot advise on that.

spring boot actuator metrics monitoring with prometheus

1 Answers