I'm trying to understand how to best use the Kubernetes readiness and liveness probes. What I have observed when testing is that if a readiness probe fails then the pod is marked as unready and removed from the load balancer. However I then expected a new pod to be started and moved into the load balancer until the original pod became ready again (or it's liveness probe failed and it is killed) and then one of them could be terminated.
We might want to temporarily remove a pod from service if the readiness probe fails but this seems to run the risk that all pods could go un-ready which would result in no pods in the load balancer and no new pods being started.
I assume what I am observing with no new pods being started to cover those that are un-ready is expected behavior? In which case what exactly is the use case for the readiness probe and how is the risk of all pods becoming unready mitigated?
