I want to add dynamic comments to metrics which gives some info on the service. Something like OS version, kernel version etc. How can I include this in the Guage metric type. I am able to add HELP and TYPE but not the OS version.
**OS version is xxxx**
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
Whatever additional info you want to slap on top of the metric, add it as dimensions (labels, in Prometheus's lingo). That way the same metrics can be used across your entire infrastructure, but you can slice them anyway you want. for example, the CPU_usage_percent metric can be enriched with the 'cloud_zone' dimension to specify which cloud it belongs to:
CPU_usage_percent{cloud_zone:”NYC”, application_name:”video-server”}, 5, 1487578310 //example of a metric from zone NYC
and then you can slice it in Prometheus with queries like:
CPU_usage_percent{cloud_zone="NYC"}
and get only the metrics you want. or their sum/average/etc. in sense, these are self-documenting.
I shamelessly point you to my article which explains this in more detail.
An somewhat alternative approach - ie rather than adding a dimension to an existing metric - you could just export these as new metrics, with bogus values (values are limited to numeric values) and encode the value in a dimension.
kernel_version 0 {ip_addr:”1.1.1.1”, version:"the actual kernel value"}
EDIT: how to add dimensions with the golang_client: refer to this code - it shows how to add labels and then populate them (in line 68)- https://github.com/prometheus/client_golang/blob/master/prometheus/examples_test.go#L51
