Kubernetes: Prometheus context deadline exceeded error

1
votes

I have several nodejs microservices which are running dev namespace which I expose metrics and able to access via http://localhost:9187/metrics.

But when I deploy prometheus server which is running monitoring namespace I received an below error in Targets page.

Get http://1.../metrics: context deadline exceeded.

I assume none of these allow access from the namespace monitoring so need to add an additional one into the namespace dev to allow the prometheus pod from namespace monitoring to scrape the below pod or what might be the reason of this error?

What is the best way to add netpol to my application to allow prometheus from namespace monitoring?

kubectl get netpol -n dev

myapp-api-dev       app.kubernetes.io/instance=myapp-api-dev,app.kubernetes.io/name=oneapihub-api           5h33m
myapp-auth-dev      app.kubernetes.io/instance=myapp-auth-dev,app.kubernetes.io/name=oneapihub-auth         56m
myapp-backend-dev   app.kubernetes.io/instance=myapp-backend-dev,app.kubernetes.io/name=oneapihub-backend   5h42m
redis               app=redis,release=redis                                                                     33d

kubectl get pods -n monitoring

NAME                                            READY   STATUS    RESTARTS   AGE
monitoring-prometheus-server-6cc796c4db-hp4rg   2/2     Running   0          2d4h
1
kubernetesprometheuskubernetes-networkpolicy
see if it helps: stackoverflow.com/questions/49817558/… – Oli

1 Answers

2
votes

I guess you have kube-prometheus installed. In this case you need to create custom roles and role bindings to let Prometheus to monitor other namespaces, see here