Openshift 4 Readiness probe failed

Question

I'm using thorntail microprofile framework to monitor a simple rest service application. The application deployment on openshift works fine but not the health monitor since receive this message:

Readiness probe failed: Get http://10.116.0.57:8080/health/live: dial tcp 10.116.0.57:8080: connect: connection refused

But can access to the health service using the service route url e.g. http://thorntail-myproject.apps-crc.testing/health/live and get the results:

{"status":"UP","checks":[{"name":"server-state","status":"UP"}]}

Both Liveness and Readiness annotations are included in the HealthCheck implementation class. Also get service response when execute curl through pod's remote container shell.

These are the dependencies I'm using in pom.xml:

<dependencies>
  <dependency>
    <groupId>io.thorntail</groupId>
    <artifactId>jaxrs</artifactId>
  </dependency>
  <dependency>
    <groupId>io.thorntail</groupId>
    <artifactId>microprofile-health</artifactId>
  </dependency>
</dependencies>

Any ideas?

I don't know, but did you try to access the URL from inside the pod? You can access the pod's terminal via the OpenShift console, or using oc rsh, and then use curl. That would be my first debugging step, anyway. — Ladicek
Already tried that and also get response from service. Question updated. — iperezmel78
Are you perhaps using Istio, or something else that messes with container networking? I remember trying to use a simple HTTP probe and failing with Istio -- had to replace it with a command based probe that simply did curl http://localhost:8080/health/.... — Ladicek
Hi @Ladicek, thanks for your replies!. I'm not using isitio, in my pom.xml only have jaxrs and microprofile-health dependencies. I have updated the question. — iperezmel78
At this point, I have no idea what could be the problem, but it seems unlikely it would be the application itself. — Ladicek

hilchev hilchev · Accepted Answer · 2020-07-28T10:43:42

The problem could be caused by many things, but here are some things you can try:

Verify that the Service object for your deployment / deploymentConfig is connecting to the correct Pods and to the correct Ports.
Verify that the Route/Ingress objects are connecting to the correct Service object.

The two things above seem correct as you can access the Route URL, but we don't know your deployments and how many you have.

Verify that your Liveness and Readiness probes are hitting the correct: Ports, page (might be a typo somewhere), protocol - are you using HTTP or HTTPS?
If all of the above is correct, check if you have additional NetworkPolicies for your namespace.

Openshift 4 Readiness probe failed

1 Answers