GCP Cloud Run: Failed to create a service

2
votes

I am trying to use Cloud run with private GKE cluster. I created the cluster using the below command:

gcloud beta container clusters create cluster-name \
    --create-subnetwork name=cloud-run-subnet \
    --enable-master-authorized-networks \
    --enable-ip-alias \
    --enable-private-nodes \
    --enable-private-endpoint \
    --master-ipv4-cidr 172.16.0.32/28 \
    --no-enable-basic-auth \
    --no-issue-client-certificate \
    --addons=HorizontalPodAutoscaling,HttpLoadBalancing,Istio,CloudRun \
    --machine-type=n1-standard-1 \
    --enable-stackdriver-kubernetes \
    --scopes cloud-platform \
    --zone us-central1-a

I created a bastion host in the same VPC and subnet that can contact this cluster using the below command:

gcloud compute instances create bastion \
 --zone us-central1-a \
 --subnet cloud-run-subnet \
 --machine-type=g1-small \
 --scopes cloud-platform

I installed kubectl on the bastion host and switched the context to this cluster using gcloud container clusters get-credentials command.

I made sure Istio is enabled on default namespace using the below command:

kubectl label namespace default istio-injection=enabled

Now when I try to deploy a service using the below command I get an error:

gcloud beta run deploy hello \
    --image=gcr.io/projectname/hello-world \
    --platform=gke \
    --cluster=cluster-name \
    --cluster-location=us-central1-a \
    --connectivity=internal

Error:

ERROR: (gcloud.beta.run.deploy) Error:                                                                                                                                                                             
failed calling webhook "webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting he
aders)

I tried with both --connectivity=internal and --connectivity=external I get the same error.

When I try to create the service using the console, I get the following error:

Failed to create a service

Tracking number: d123456789

I have made sure to keep the same project for the container registry (gcr.io) and GKE cluster.

Exact things work fine with public clusters. I am not able to find specific documentation to get it all working with private GKE cluster. Is there anything that I am missing? Is there anything that I am doing wrong?

2
google-cloud-platformgoogle-kubernetes-engineistiogoogle-cloud-runknative

2 Answers

1
votes

A similar issue was fixed doing that in this GitHub thread:

https://github.com/knative/serving/issues/4868

The main cause is that in a private GKE cluster, by default only the GKE master have access to the services at port 443 or 80.

Could you try to use the port 8443 instead of the 443, and white-list the port 8443.

https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules

0
votes

Can you try to deploy Cloud Run service with yaml file?

Here the file

apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: hello
  namespace: default
spec:
  traffic:
  - percent: 100
    latestRevision: true
  template:
    spec:
      containers:
      - image: gcr.io/projectname/hello-world

Simply perform a kubectl apply -f <file> from your bastion

If it works, this means that gcloud command can't communicate with a private cluster (and you can open an issue on this).