I am trying to figure, how to get pod labels into the metric tags from kubelet metrics using prometheus-stack. In our environment, we need to hash pod names (due to length limitations) so our app name, env, and unit name are saved in pod labels.
We are using prometheus-stack (helm installation) to collect metrics from kubelet (/metrics, /metrics/cadvisor) and due to the lack of pod labels in metrics tags, it's difficult to know which metric belongs to which application.
Prometheus-stack is using sd_kubernetes_config with endpoint rule to collect kubelet metrics, where __meta tags for pod labels cannot be used. Is there another way how to get that labels in metric tags?
I also tried to collect pod_labels metric using kubeStateMetrics, where I can get metric that contains pod labels, but I cannot figure out how to display both metrics in a way that metric from cadvisor will show its value and metric from kubeStateMetrics will be used to display its labels (in prometheus graph).
Thanks for any advice.
