Readiness probe failed but still running and endPoint doesn't remove the pod id

1
votes

I describe the pod. I found the error message. It seems the pod Readiness probe is failed. But I check the endpoint, the endpoint doesn't remove the pod IP. The pod receive traffic, even the liveness probe is failed. 

  Warning  Unhealthy  92s (x84 over 3d1h)  kubelet, aks-agentpool-xxxxx Readiness probe failed: Get https://10.244.14.21:5001/api/alive: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

$ kubectl describe ep myapp
Name:         myapp
Namespace:    default
Labels:       <none>
Annotations:  <none>
Subsets:
  Addresses:          10.244.14.21,10.244.24.34
  NotReadyAddresses:  <none>
  Ports:
    Name    Port  Protocol
    ----    ----  --------
    legacy  5001  TCP
    https   5001  TCP

Events:  <none>

I want to know what if readiness failed?

1
kubernetes
Sometimes, we need a pod's ip appearing in the endpoint list even it's not ready yet, in that way, we will set publishNotReadyAddresses: true in the service yaml. You can check if it's that case. – Kun Li
I don't set publishNotReadyAddresses. – Ruoxuan.Wang
Then, from the time of the event, the 'Readiness probe failed' event happened '92s' ago, maybe it become healthy after that warning, you can check that by mannual. – Kun Li
is it still actual? – Nick
No. The kubernetes endPoint works well. I misunderstand the kubelet log. – Ruoxuan.Wang

1 Answers

1
votes

Practice

It seems the pod Readiness probe is failed. But I check the endpoint, the endpoint doesn't remove the pod IP.

The Pod is healthy at the time you are checking it and the Warning you see is just saying that there were situations when Readiness probe failed for it.

Please note that if the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod (they are added back upon Readiness probe success).

You can check that with $ kubectl describe svc/YOUR_SERVICE .

It'll output the list of EndPoints (EPs) that serve traffic for that service at a time you executed command.

If the EP readiness probe fails, EP won't be listed in the output of that command.

From the output you have provided:

$ kubectl describe ep myapp
...
Subsets:
  Addresses:          10.244.14.21,10.244.24.34
  NotReadyAddresses:  <none>

we can say that all the EPs are ready at the moment.

Theory

I want to know what if readiness failed?

In short: traffic will be redirected to other EPs untill this EP is ready again.

Official documentation on Readiness probe says that:

A Probe is a diagnostic performed periodically by the kubelet on a Container. To perform a diagnostic, the kubelet calls a Handler implemented by the Container.

The kubelet can optionally perform and react to three kinds of probes on running Containers: - livenessProbe - readinessProbe - startupProbe

readinessProbe: Indicates whether the Container is ready to service requests. If the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod. The default state of readiness before the initial delay is Failure. If a Container does not provide a readiness probe, the default state is Success.

Hope that is exactly the info you have been looking for.