Prometheus add custom metrics

3
votes

We have a Prometheus running in our cluster and we are able to use grafana to watch our cluster / pods metrics, now I want to add some custom metrics , is there a way to do it ? if so how should I connect the code to Prometheus , I mean if i write golang program using Prometheus API , and deploy it as docker to k8s , now does the program know to connect with Prometheus ? e.g. this program is exposing data to the /metrics endpoint but what else should I do to make prom to be able to read this data ?

https://gist.github.com/sysdig-blog/3640f39a7bb1172f986d0e2080c64a75#file-prometheus-metrics-golang-go

1
gokubernetesprometheus
If you add your custom metrics to your program, give them names you can recognize, and then expose them via an http handler like it is done in that example, it should be enough. Your program does not push data to Prometheus, it will scrape data from your http metrics endpoint, and if you publish your metrics there, they should show up in Prometheus next time it scrapes your application. – Burak Serdar
@bserdar - just for verification, so I can deploy the program to k8s as-is and I be able to query it by prom ui ? thanks! – Jhon D
If your Prometheus instance is scraping pods, yes. You can verify if you're publishing your metrics by simply looking at your app's metrics endpoint. It is text, and you should see your metrics there. – Burak Serdar
@bserdar - Ok i'll deploy my app to k8s and check it, thank you! – Jhon D
To allow prometheus to collect (scrap) your pods, you might want to look at k8sClusterRoles see here: supergiant.io/blog/… – colm.anseo

1 Answers

1
votes

You probably want Prometheus to discover the targets to scrape from automatically, if so you should configure K8s service discovery in Prometheus to discover pods, nodes and services from your cluster (maybe you already did something like this since you are already monitoring k8s metrics).

To get your go application monitored you can for example add annotations to your pods or to your services to enable scraping from this targets and define where the metrics are available (path, port). However this depends on your scrape configuration and relabeling. Good example can be found here

If you are using Prometheus Operator you need to define a servicemonitor resource instead.