How to leverage Kubernetes readinessProbe to self heal the pod without restarting it?

0
votes

According to this documentation, I see that readinessProbe can be used to temporarily halt requests to a pod without having to restart it in order to recover gracefully.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-readiness-probes

When I see pod events it looks like the pod is restarted upon Readiness probe failure. Here are the events:

 1. Readiness probe failed
 2. Created container
 3. Started container
 4. Killing container with id {}

Tried to modify container restartPolicy to OnFailure hoping this configuration decides pod action upon readinessProbe failure but I see the following error:

The Deployment {} is invalid: spec.template.spec.restartPolicy: Unsupported value: "OnFailure": supported values: "Always"

Which is the right way to stop requests to a pod without having to restart it and letting the application gracefully recover?

1
kuberneteskubernetes-health-check
Can you share the POD yaml? – Crou

1 Answers

0
votes

There are two type of probes. Restarts happen due to failing liveness probes. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

liveness probe

The kubelet uses liveness probes to know when to restart a Container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a Container in such a state can help to make the application more available despite bugs.

readiness probe

Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup, or depend on external services after startup. In such cases, you don’t want to kill the application, but you don’t want to send it requests either. Kubernetes provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.

Today I found a very good essay about probes https://blog.colinbreck.com/kubernetes-liveness-and-readiness-probes-how-to-avoid-shooting-yourself-in-the-foot/