1
votes

I have an application that uses GKE Ingress (master version 1.10.6-gke.2) for a load balancer. Recently GKE started supporting declaring IAP support via BackendConfig. I followed the documentation at [1] and [2]. However, now, GKE seems to hang while creating my Ingress.

Below is the yaml for my service, ingress and backendconfig.

kubectl -n randall-test-1 get svc,ing,backendconfig -o yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      beta.cloud.google.com/backend-config: '{"default": "airflow-backend-config"}'
      service.alpha.kubernetes.io/app-protocols: '{"web":"HTTPS"}'
    creationTimestamp: 2018-09-10T19:23:13Z
    name: airflow
    namespace: randall-test-1
    resourceVersion: "2155724"
    selfLink: /api/v1/namespaces/randall-test-1/services/airflow
    uid: X-X-X-X-X
  spec:
    clusterIP: X.X.X.X
    externalTrafficPolicy: Cluster
    ports:
    - name: web
      nodePort: 30099
      port: 8080
      protocol: TCP
      targetPort: web
    selector:
      app: airflow
    sessionAffinity: None
    type: NodePort
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.allow-http: "false"
    creationTimestamp: 2018-09-10T19:23:13Z
    generation: 1
    name: airflow
    namespace: randall-test-1
    resourceVersion: "2155721"
    selfLink: /apis/extensions/v1beta1/namespaces/randall-test-1/ingresses/airflow
    uid: X-X-X-X-X
  spec:
    backend:
      serviceName: airflow
      servicePort: 8080
    tls:
    - secretName: tls
  status:
    loadBalancer: {}
- apiVersion: cloud.google.com/v1beta1
  kind: BackendConfig
  metadata:
    clusterName: ""
    creationTimestamp: 2018-09-10T19:23:13Z
    generation: 1
    name: airflow-backend-config
    namespace: randall-test-1
    resourceVersion: "2155728"
    selfLink: /apis/cloud.google.com/v1beta1/namespaces/randall-test-1/backendconfigs/airflow-backend-config
    uid: X-X-X-X-X
  spec:
    iap:
      enabled: true
      oauthclientCredentials:
        secretName: oauth2
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The hang gives me no insight.

cluster@master0:~/kube-config$ kubectl -n randall-test-1 describe ing
Name:             airflow
Namespace:        randall-test-1
Address:
Default backend:  airflow:8080 (X.X.X.X:8080)
TLS:
  tls terminates
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     airflow:8080 (X.X.X.X:8080)
Annotations:
Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     6m    loadbalancer-controller  randall-test-1/airflow

However, in GKE console, I just get Creating ingress as a status for > 20 mins with no resolution. I also check my Load Balancers in console and see nothing.

Any ideas what is happening or what else I can check?

I also tried to do this with just securityPolicy which is supposed to link the Load Balancer with a Cloud Armor policy. This also doesn't work with a similar hang.

[1] https://cloud.google.com/iap/docs/enabling-kubernetes-howto

[2] https://cloud.google.com/kubernetes-engine/docs/concepts/backendconfig

NOTE: cross posted at https://github.com/kubernetes/ingress-gce/issues/469

1
did you check the logs for kube-controller-manager and kube-apiserver in your active k8s master. Some auth or api to GCP might be failing.Rico

1 Answers

0
votes

We've been getting a few similar cases in the past few days. There seems to be something wrong with permissions of the default GKE service account.

Can you try to add the following permissions to it:

  • clientauthconfig.clients.update
  • clientauthconfig.clients.get