When Prometheus uses Consul's auto-discovery feature to get the list of targets to monitor, it also gets Consul servers themselves. This is great - we want to monitor these guys with Prometheus. The problem is that Consul reports these nodes with port 8300, which is not the port we use to monitor targets.
How does one replace ports received from Consul to something else? I know this is possible with Prometheus relabel_configs but I've yet to successfully configure it.
I've eventually figured it out. below is a working example. As the documentation specifies, the address keyword might not work for all setups - you might want to try "<__meta_consul_address>:<__meta_consul_service_port>" instead.
- source_labels: ['__address__']
separator: ':'
regex: '(.*):(8300)'
target_label: '__address__'
replacement: '${1}:9126'
In case you would like to dynamically define port by additional annotation, ie.:
selector:
matchLabels:
app: my-service
template:
metadata:
labels:
app: my-service
annotations:
prometheus: "true"
prometheus/port: "8888"
You can use the following transformation:
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: ${1}:${2}
target_label: __address__
Source labels in prometheus are joined with ; and in this case we will have ie. (if your server by default listening on 8080): http://10.52.9.79:8080;8888
In regex we have:
:, ie.: //10.52.9.79As a result you set original address and port from annotation. Could be useful if for example you have Spring Boot application and you want to use different management port than default application port.
