1
votes

We are running the node-exporter in containers. To quickly identify on which host each node-exporter is running, I created a metric that looks like this: host{host="$HOSTNAME",node="$CONTAINER_ID"} 1

I'm looking for a way to extract the hostname in host= and create a label for each node-exporter instance as a hostname label. I tried numerous configurations and none seem to work. Current prometheus config looks like this:

scrape_configs:
   - job_name: 'node'
     scrape_interval: 10s
     scrape_timeout: 5s
     metrics_path: /metrics
     scheme: http
     dns_sd_configs:
     - names:
     - tasks.master-nodeexporter
       refresh_interval: 30s
       type: A
       port: 9100
     relabel_configs:
     - source_labels: ['host']
       regex: '"(.*)".*'
       target_label: 'hostname'
       replacement: '$1'
2

2 Answers

3
votes

This is not possible, as target relabelling happens before the scrape.

What you want to do here is use service discovery to have the right hostname in the first place, which is not possible with dns_sd_configs. You might look at something like Consul and https://www.robustperception.io/controlling-the-instance-label/

0
votes

If someone comes across this:

Create this as docker-entrypoint.sh and make it executable.

#!/bin/sh -e
# Must be executable by others

NODE_NAME=$(cat /etc/nodename) echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom

set -- /bin/node_exporter "$@"

exec "$@"

Than create a Dockerfile like this

FROM prom/node-exporter:latest

ENV NODE_ID=none

USER root

COPY conf /etc/node-exporter/

ENTRYPOINT [ "/etc/node-exporter/docker-entrypoint.sh" ]
CMD [ "/bin/node_exporter" ]

Than build it and you will always get the Hostname as a node_meta metric