Can prometheus find alert manager dynamically with out restart itself

Question

I run prometheus and alertmanager on pods in k8s cluster, I want prometheus can find alertmanager dynamically, when alertmanager pod start it add it to it's config file without restart itself, because if prometheus stop it means the pod that host prometheus ends, is there any way to do it?

Answer 1

Prometheus locates the alertmanager via the alertmanager config inside the prometheus config file. When deployed on Kubernetes, the alertmanager pod can start on a new node and thus using an IP address is not a reliable way to point prometheus to the alertmanager.

The recommended way in Kubernetes is to create a service for the alertmanager and use the service DNS name inside the prometheus.yml file. That way prometheus will always be able the alertmanager even when the alertmanager pod restarts on a different node.