Liveness and Readiness probes failing in Kubernetes cluster- istio proxy sidecar injection is enabled in application

1
votes

Below is the config for probes in my application helm chart

{{- if .Values.endpoint.liveness }}
          livenessProbe:
            httpGet:
              host: localhost
              path: {{ .Values.endpoint.liveness | quote }}
              port: 9080
            initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
            periodSeconds: 5
{{- end }}
{{- if .Values.endpoint.readiness }}
          readinessProbe:
            httpGet:
              host: localhost
              path: {{ .Values.endpoint.readiness | quote }}
              port: 9080
            initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
            periodSeconds: 60
{{- end }}
{{- end }}

when I deploy, in deployment.yaml

livenessProbe:
            httpGet:
              path: /my/app/path/health
              port: 9080
              host: localhost
              scheme: HTTP
            initialDelaySeconds: 8
            timeoutSeconds: 1
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /my/app/path/health
              port: 9080
              host: localhost
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 60
            successThreshold: 1
            failureThreshold: 3

But in pod.yaml, it is

livenessProbe:
       httpGet:
         path: /app-health/app-name/livez
         port: 15020
         host: localhost
         scheme: HTTP
       initialDelaySeconds: 8
       timeoutSeconds: 1
       periodSeconds: 5
       successThreshold: 1
       failureThreshold: 3
     readinessProbe:
       httpGet:
         path: /app-health/app-name/readyz
         port: 15020
         host: localhost
         scheme: HTTP
       initialDelaySeconds: 5
       timeoutSeconds: 1
       periodSeconds: 60
       successThreshold: 1
       failureThreshold: 3

and then gives the following error in the pod:

`Readiness probe failed: Get http://IP:15021/healthz/ready: dial tcp IP:15021: connect: connection refused spec.containers{istio-proxy}

warning Liveness probe failed: Get http://localhost:15020/app-health/app-name/livez: dial tcp 127.0.0.1:15020: connect: connection refused spec.containers{app-name}

warning Readiness probe failed: Get http://localhost:15020/app-health/app-name/readyz: dial tcp 127.0.0.1:15020: connect: connection refused spec.containers{app-name} `

why is the pod using a different path and port for the probes and it is failing giving the above error. Can someone please help me with what am missing?

1
kubernetesistioistio-sidecarreadinessprobelivenessprobe
You can find the documentation about healthchecks in Istio by following this link: istio.io/latest/docs/ops/configuration/mesh/app-health-check. Have you tried the part about the: Disable the HTTP probe rewrite for a pod? – Dawid Kruk

1 Answers

1
votes

You're getting those different paths because those are globally configured across mesh in Istio's control plane component i.e., istio-sidecar-injector configmap This is coming via sidecar's webhook injection. See for the below property in "istio-sidecar-injector configmap"

sidecarInjectorWebhook.rewriteAppHTTPProbe=true