grafana dashboard for prometheus not working

2
votes

I am newbie to grafana and prometheus. I setup prometheus, grafana, alertmanager, nodeexporter and cadvisor using the docker-compose.yml from this post https://github.com/vegasbrianc/prometheus

And imported grafana dashboard #893 from https://grafana.com/dashboards/893

But the dashboard is not working as I can see N/A in some panels. For example below are the queries used by the panels and I couldn't figure out how to get the values for the template variable in the query. I looked at http://node-exporter:9100/metrics and do not see a value for variable '$server'

Query1: time() - node_boot_time{instance=~"$server:.*"}
Query2:min((node_filesystem_size_bytes{fstype=~"xfs|ext4",instance=~"$server:.*"} - node_filesystem_free_bytes{fstype=~"xfs|ext4",instance=~"$server:.*"} )/ node_filesystem_size_bytes{fstype=~"xfs|ext4",instance=~"$server:.*"})

What should I configure for node-exporter and prometheus to evaluate the template variable $server in the queries?

2
grafanaprometheus
Also in my dashboard settings --> variables i see a variable 'server' of type 'query' and query=label_values(node_boot_time, instance). How does this query work – cnu

2 Answers

1
votes

$server is a Grafana template variable. These usually show up as dropdowns at the top of the Grafana dashboard.

label_values is a Prometheus-specific Grafana function that is applied to a Prometheus query. Your particular example, label_values(node_boot_time, instance) will return all values of the instance label for all node_boot_time metrics collected by Prometheus (i.e. all node exporter targets monitored by Prometheus).

I have no experience with the particular dashboard you are using (or node exporter, for that matter), but usually the cause for some panels displaying "N/A" or no values while other panels work just fine is that the underlying metric names might have changed. You can click on the header of the problematic panel in Grafana, select Edit, then click on the Metrics tab to try different metric names. For "inspiration", check the /metrics endpoint of your node exporter. If you don't know how to get to it, on the Prometheus web interface navigate to Status > Targets and click on the URL of your node exporter.

0
votes

An old question, but it still didn't work for me. The reason is that the label_values(...) works fine obtaining all the instance names that have a node_boot_time metric.

The problem is in the regex that follows the expression (next line). In my case it was something tricky resembling "/([^:].*):/". My instance names start with "i-" and contain no colon, so nothing was being selected. I just used a ProductCode to figure out the right instances instead.