How to config email alert in using grafana and prometheus-operator

Question

I installed prometheus-operator (include prometheus/alertmanager/grafana) via helm. Then I access Grafana UI and config alert via email. When I click send an email test, I got the message “ SMTP not configured, check your grafana.ini config file’s [smtp] section”

But I don’t know where the grafana.ini to can change in this case.

[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5bbc8f45cb-nlqgh                  1/1     Running   1          15h
kube-system   calico-node-lk2j5                                         1/1     Running   1          15h
kube-system   calico-node-v6wzs                                         1/1     Running   1          15h
kube-system   calico-node-zfh5r                                         1/1     Running   1          15h
kube-system   coredns-5c98db65d4-79c2g                                  1/1     Running   1          15h
kube-system   coredns-5c98db65d4-bqj7g                                  1/1     Running   1          15h
kube-system   etcd-k8s-master                                           1/1     Running   1          15h
kube-system   kube-apiserver-k8s-master                                 1/1     Running   1          15h
kube-system   kube-controller-manager-k8s-master                        1/1     Running   2          15h
kube-system   kube-proxy-8qmdt                                          1/1     Running   1          15h
kube-system   kube-proxy-qwgbc                                          1/1     Running   1          15h
kube-system   kube-proxy-vhqjd                                          1/1     Running   1          15h
kube-system   kube-scheduler-k8s-master                                 1/1     Running   1          15h
monitoring    alertmanager-prometheus-operator-alertmanager-0           2/2     Running   3          15h
monitoring    prometheus-operator-grafana-64848fc9bb-dbnwc              2/2     Running   3          15h
monitoring    prometheus-operator-kube-state-metrics-5d46566c59-ck4np   1/1     Running   2          15h
monitoring    prometheus-operator-operator-64dcc7bfc-lpdj6              2/2     Running   2          15h
monitoring    prometheus-operator-prometheus-node-exporter-ns4kg        1/1     Running   1          15h
monitoring    prometheus-operator-prometheus-node-exporter-tdhwq        1/1     Running   2          15h
monitoring    prometheus-operator-prometheus-node-exporter-xt8z9        1/1     Running   2          15h
monitoring    prometheus-prometheus-operator-prometheus-0               3/3     Running   4          15h

Answer 1

You will be able to override this configuration, using helm variables, thanks to alertmanager.config key. This key convert yaml into configuration for alertmanager, so you can use every alertmanager configuration. You should probably also change grafana.ini configuration to configure smtp into grafana (test seems to use that configuration). You can check this configuration in Grafana via "Server admin" > "Settings", search "smtp".

As a reference, you can do something like the following for alertmanager :

    helm upgrade --install prometheus stable/prometheus-operator \
-f helm/prometheus-operator.yml \
-f helm/grafana-custom.staging.yml \
--set-string alertmanager.config.global.smtp_smarthost="my.smtp.tld:465" \
--set-string alertmanager.config.global.smtp_auth_username="[email protected]" \
--set-string alertmanager.config.global.smtp_from="[email protected]" \
--set-string alertmanager.config.global.smtp_auth_password="MyAmazingPassword" \
--set-string grafana.'grafana\.ini'.smtp.enabled=true \
--set-string grafana.'grafana\.ini'.smtp.host="my.smtp.tld:465" \
--set-string grafana.'grafana\.ini'.smtp.from_address="[email protected]" \
--set-string grafana.'grafana\.ini'.smtp.user="[email protected]" \
--set-string grafana.'grafana\.ini'.smtp.password="MyAmazingPassword"

Answer 2

The grafana.ini is loaded through configmaps in prometheus-operator helm deployment. If you have already installed it via helm then you can just modify the configmap and then restart the grafana pod. Below is minimum config with which i was able to use SMTP.

[smtp]
enabled = true
host = your.smtp.server.name:25
skip_verify = true
from_address = "[email protected]"
from_name = Grafana

To get the configmap, run below command and edit the configmap(include namespace in below command if prometheus-operator is deployed in a speratae name space than default).

kubectl get configmap | grep grafana

After editing configmap, restart grafana pod(restarts of other pods is no needed).

Note: Skip_verify = true is not recommended.