GCE Ingress not picking up health check from readiness probe

11
votes

When I create a GCE ingress, Google Load Balancer does not set the health check from the readiness probe. According to the docs (Ingress GCE health checks) it should pick it up.

Expose an arbitrary URL as a readiness probe on the pods backing the Service.

Any ideas why?

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: frontend-prod
  labels:
    app: frontend-prod
spec:
  selector:
    matchLabels:
      app: frontend-prod
  replicas: 3
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: frontend-prod
    spec:
      imagePullSecrets:
        - name: regcred
      containers:
      - image: app:latest
        readinessProbe:
          httpGet:
            path: /healthcheck
            port: 3000
          initialDelaySeconds: 15
          periodSeconds: 5
        name: frontend-prod-app
      - env:
        - name: PASSWORD_PROTECT
          value: "1"
        image: nginx:latest
        readinessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
        name: frontend-prod-nginx

Service:

apiVersion: v1
kind: Service
metadata:
  name: frontend-prod
  labels:
    app: frontend-prod
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: frontend-prod

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: frontend-prod-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: frontend-prod-ip
spec:
  tls:
    - secretName: testsecret
  backend:
    serviceName: frontend-prod
    servicePort: 80
5
kubernetesgoogle-compute-enginegoogle-kubernetes-enginekubernetes-health-check
Do you have anything serving on http://:3000/healthcheck and http://:80/health? – suren
Yes they exist and the checks are passing in kubernetes. – Amarjeet Singh Rai
There are some limitations here[1]. I am going to reproduce your use case. [1]: github.com/kubernetes/ingress-gce/blob/master/examples/… – suren
Awesome. I'm using GKE with latest version of kubernetes they provide no alpha features. – Amarjeet Singh Rai
Hi there. So, I did the test, and it worked. Now, when describing the ingress, there is nothing there. I went to the LB page > health checks, in GCP and there is was; a note saying "Kubernetes L7 health check generated with readiness probe settings." I made the test with one nginx container though. May be it is necessary to run a test with two containers; one of them on the service port, the other one on random port. – suren

5 Answers

13
votes

So apparently, you need to include the container port on the PodSpec. Does not seem to be documented anywhere.

e.g.

    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Thank you to Brian! https://github.com/kubernetes/ingress-gce/issues/241

4
votes

This is now possible in the latest GKE (I am on 1.14.10-gke.27, not sure if that matters)

  1. Define a readinessProbe on your container in your Deployment.
  2. Recreate your Ingress.
  3. The health check will point to the path in readinessProbe.httpGet.path of the Deployment yaml config.
2
votes

Update by Jonathan Lin below: This has been fixed very recently. Define a readinessProbe on the Deployment. Recreate your Ingress. It will pick up the health check path from the readinessProbe.

GKE Ingress health check path is currently not configurable. You can go to http://console.cloud.google.com (UI) and visit Load Balancers list to see the health check it uses.

Currently the health check for an Ingress is GET / on each backend: specified on the Ingress. So all your apps behind a GKE Ingress must return HTTP 200 OK to GET / requests.

That said, the health checks you specified on your Pods are still being used ––by the kubelet to make sure your Pod is actually functioning and healthy.

2
votes

Google has recently added support for CRD that can configure your Backend Services along with healthchecks:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: backend-config
  namespace: prod
spec:
  healthCheck:
    checkIntervalSec: 30
    port: 8080
    type: HTTP #case-sensitive
    requestPath: /healthcheck

See here.

0
votes

Another reason why Google Cloud Load Balancer does not pick-up GCE health check configuration from Kubernetes Pod readiness probe could be that the service is configured as "selectorless" (the selector attribute is empty and you manage endpoints directly).

This is the case with e.g. kube-lego: see https://github.com/jetstack/kube-lego/issues/68#issuecomment-303748457 and https://github.com/jetstack/kube-lego/issues/68#issuecomment-327457982.

Original question does have selector specified in the service, so this hint doesn't apply. This hints serves visitors that have the same problem with a different cause.