prometheus-operator (helm chart) & alert manager

3
votes

I have a query related with prometheus-operator helm chart & alert manager combination.

Currently we are using prometheus-operator helm chart: https://github.com/helm/charts/tree/master/stable/prometheus-operator

and I wrote a simple rule in values.yml (this is just a sample code) to generate an alert:

enter image description here

further I am using alertmanager config/routes/receivers to send alerts. It's working perfectly fine.

But as part of real-time implementation, I may be having so many alert rules. Is there any way where I can bring these all rules in separate rules file & configure the path (rule file path) in values.yml (under: additionalPrometheusRules section)

I also saw kube-prometheus-stack & additionalPrometheusRulesMap (in values.yml): https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

But didn't fine any solution. Anyone can help me on this?

2
kubernetes-helmprometheus-operator

2 Answers

0
votes

So helm doesn't allow includes in values.yaml files typically. I read that there's a way to do it, but it depends on how the chart is built and typically upstream maintainers don't use templates that way afaik (could be wrong there, but I've never noticed it).

Your problem is exactly the same problem I've been trying to solve adequately, and I think I came up with something. It's not perfect, but it is better than having one huge monolithic values.yaml file.

helm allows the operator to specify multiple values.yaml files using the paradigm, -f values1.yaml -f values2.yaml -f some-more-values.yaml, so I broke my values file up into multiple logically-divided yaml files.

There might be gotchas, so be aware, but so far for this use-case, it seems to be working. I'm still testing things out. https://helm.sh/docs/helm/helm_install/

0
votes

You can also add your own custom rules file using config maps. In this way, you can avoid over alerting and get notified for specific alerts only.