can't evaluate field instance in type struct : AlertManager

Question

I have configured an alert on when CloudWatch exporter is down using AlertManager server (which in installed in same server as Prometheus). The rule is the following:

groups:
- name: Alerts
  rules:

      # Alert for any instance that is unreachable for >5 minutes.
      - alert: CloudWatchExporterDown
        expr: up{instance="localhost:9106",job="cloudwatch_exporter"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ .instance }} down"
          description: "{{ .instance }} of job {{ .job }} has been down for more than 5 minutes."

Now I have in /var/log/messages these errors :

Sep 21 03:55:50 ip-10-193-192-40 prometheus: level=warn ts=2020-09-21T03:55:50.728Z caller=alerting.go:343 component="rule manager" alert=CloudWatchExporterDown msg="Expanding alert template failed" err="error executing template __alert_CloudWatchExporterDown: template: __alert_CloudWatchExporterDown:1:92: executing \"__alert_CloudWatchExporterDown\" at <.instance>: can't evaluate field instance in type struct { Labels map[string]string; ExternalLabels map[string]string; Value float64 }" data="unsupported value type"

I would like to know what is wrong in the rule ? Why the expression { .instance } is not evaluated?

Michael Doubez Michael Doubez · Accepted Answer · 2020-09-23T10:10:45

Prometheus is using Go language templating:

Label and annotation values can be templated using console templates. The $labels variable holds the label key/value pairs of an alert instance. The configured external labels can be accessed via the $externalLabels variable. The $value variable holds the evaluated value of an alert instance.

From the documentation and the examples you can find, the correct expression should use $labels:

summary: "Instance {{ $labels.instance }} down"

can't evaluate field instance in type struct : AlertManager

1 Answers