Kubernetes Readiness probe failed error

27

votes

While running my container on kubernetes using helm upgrade command, I am getting this error:

'Readiness probe failed: Get http://172.17.0.6:3003/: dial tcp 172.17.0.6:3003: getsockopt: connection refused'.

My docker image is for a node.js application and I am trying to manage it through minikube.

21

votes

For anyone else here, if using helm to manage your deployments, you need to set initialDelaySeconds it in the deployments.yaml template in the /templates folder under livenessProbe. The livenessProbe will force restart your pod if the probe cannot connect, as was happening with mine. It wasn't giving my application enough time to build.

10

votes

This could be solved by increasing the initial delay in the readiness check. Actually since the connection to the DB was taking more then the initial delay as a result of which the readiness probe was failing.

4

votes

Helm:

I would recommend setting the initialDelaySeconds value in the values.yaml file and use an action {{ .Values.initialDelaySeconds }} to insert the value into the deployment.yaml template.

kubectl:

Just add the initialDelaySeconds: 5 if you want 5 seconds to your (deployment,pod,replicate set etc) manifest and apply your changes.

If it fails, get coffee and start looking at logs from the container

kubectl logs -h for more help

2

votes

Update coredns image from v1.5.0 to current version v1.6.9, then the error got fixed.

1

votes

I had this issue too. It was fixed by specifying that my docker image listen on host 0.0.0.0 with the command set in my dockerfile: ENV HOST '0.0.0.0'.

When deploying to a Docker, and potentially other, containers, it is advisable to listen on 0.0.0.0 because they do not default to exposing mapped ports to localhost.

0

votes

Helm & NodeJs :

For my Node.js app, 5s of initialDelaySeconds wasn't the exact solution.

I solved the issue by using this health-connect library from @cloudnative.

Just follow the instruction (README) to put some code importing the package and setting the app to respond to the liveness and readiness check request.

And make sure you write the correct path and port for both livenessProbe and readinessProbe in /templates - deployment.yaml.

  livenessProbe:
     initialDelaySeconds: {{ .Values.initialDelaySeconds }}
     httpGet:
       path: /live
       port: 3000
  readinessProbe:
     initialDelaySeconds: {{ .Values.initialDelaySeconds }}
     httpGet:
       path: /ready
       port: 3000

reference: https://developer.ibm.com/tutorials/health-checking-kubernetes-nodejs-application/

Kubernetes Readiness probe failed error

6 Answers