Access Kubernetes GKE cluster outside of GKE cluster with client-go?

4
votes
  • I have multiple kubernetes clusters running on GKE (let's say clusterA and clusterB)
  • I want to access both of those clusters from client-go in an app that is running in one of those clusters (e.g. access clusterB from an app that is running on clusterA)

I general for authenticating with kubernetes clusters from client-go I see that I have two options:

  • InCluster config
  • or from kube config file

So it is easy to access clusterA from clusterA but not clusterB from clusterA.

What are my options here? It seems that I just cannot pass GOOGLE_APPLICATION_CREDENTIALS and hope that client-go will take care of itself.

So my thinking:

  • create a dedicated IAM service account
  • create kube config with tokens for both clusters by doing gcloud container clusters get-credentials clusterA and gcloud container clusters get-credentials clusterB
  • use that kube config file in client-go via BuildConfigFromFlags on clusterA

Is this the correct approach, or is there a simpler way? I see that tokens have an expiration date?

Update:

It seems I can also use CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True gcloud beta container clusters get-credentials clusterB --zone. Which would add certificates to kube conf which I could use. But AFAIK those certificates cannot be revoked

1
kubernetesgoogle-kubernetes-engine

1 Answers

10
votes

client-go needs to know about:

  1. cluster master’s IP address
  2. cluster’s CA certificate

(If you're using GKE, you can see these info in $HOME/.kube/config, populated by gcloud container clusters get-credentials command).

I recommend you to either:

  1. Have a kubeconfig file that contains these info for clusters A & B
  2. Use GKE API to retrieve these info for clusters A & B (example here) (You'll need a service account to do this, explained below.)

Once you can create a *rest.Config object in client-go, client-go will use the auth plugin that's specified in the kubeconfig file (or its in-memory equivalent you constructed). In gcp auth plugin, it knows how to retrieve a token.

Then, Create a Cloud IAM Service Account and give it "Container Developer" role. Download its key.

Now, you have two options:

Option 1: your program uses gcloud

gcloud auth activate-service-account --key-file=key.json
KUBECONFIG=a.yaml gcloud container clusters get-credentials clusterA
KUBECONFIG=b.yaml gcloud container clusters get-credentials clusterB

Then create 2 different *rest.Client objects, one created from a.yaml, another from b.yaml in your program.

Now your program will rely on gcloud binary to retrieve token every time your token expires (every 1 hour).

Option 2: use GOOGLE_APPLICATION_CREDENTIALS

  1. Don't install gcloud to your program’s environment.
  2. Set your key.json to GOOGLE_APPLICATION_CREDENTIALS environment variable for your program.
  3. Figure out a way to get cluster IP/CA (explained above) so you can construct two different *rest.Config objects for cluster A & B.
  4. Now your program will use the specified key file to get an access_token to Google API every time it expires (every 1h).

Hope this helps.

P.S. do not forget to import _ "k8s.io/client-go/plugin/pkg/client/auth/gcp" in your Go program. This loads the gcp auth plugin!