Service Account Unable to Interact with GKE Cluster in Another Project

Question

A service account ($SERVICE_ACCOUNT_A) from one Google Cloud Platform (GCP) project ($PROJECT_A) is unable to interact with a Google Kubernetes Engine (GKE) cluster ($GKE_CLUSTER_B) within another GCP project ($PROJECT_B); where:

$PROJECT_A is the name of the project $SERVICE_ACCOUNT_A lives within
$SERVICE_ACCOUNT_A is of the form some-name@some-project-name@.iam.gserviceaccount.com
$PROJECT_B is the name of the project the $GKE_CLUSTER_B cluster lives within
$GKE_CLUSTER_B is a GKE cluster name, not context, of the form: some_cluster

$SERVICE_ACCOUNT_A is unable to interact with a $GKE_CLUSTER_B despite possessing roles from $PROJECT_B containing permissions that should allow it to do so.

I.e., first I created a custom role $ROLE:

gcloud iam roles create $ROLE \
--description="$ROLE_DESCRIPTION" \
--permissions=container.clusters.get,container.clusters.list \
--project=$PROJECT_B \
--title='$ROLE_TITLE'

#=>

Created role [$ROLE].
description: $ROLE_DESCRIPTION
etag: . . .
includedPermissions:
- container.clusters.get
- container.clusters.list
name: projects/$PROJECT_B/roles/$ROLE
stage: . . .
title: $ROLE_TITLE

then I associated $ROLE, from $PROJECT_B, with $SERVICE_ACCOUNT_A:

gcloud projects add-iam-policy-binding $PROJECT_B \
--member=serviceAccount:$SERVICE_ACCOUNT_A \
--role=projects/$PROJECT_B/roles/$ROLE

#=>

Updated IAM policy for project [$PROJECT_B].
auditConfigs:
. . .

and I am able to see $ROLE under $SERVICE_ACCOUNT_A:

gcloud projects get-iam-policy $PROJECT_B \
--flatten='bindings[].members' \
--format='value(bindings.role)' \
--filter="bindings.members:${SERVICE_ACCOUNT_A}"

#=>

projects/$PROJECT_B/roles/$ROLE

with the proper permissions:

gcloud iam roles describe $ROLE \
--flatten='includedPermissions' \
--format='value(includedPermissions)' \
--project=$PROJECT_B

#=>

container.clusters.get
container.clusters.list

but still unable to get $SERVICE_ACCOUNT_A to interact with $GKE_CLUSTER_B.

Why?

Mike Mike · Accepted Answer · 2021-07-01T02:22:29

You need to enable the Kubernetes Engine API (found here) for $PROJECT_A, even if $PROJECT_A doesn't have or need a GKE cluster.

You can confirm this by creating a new JSON key for $SERVICE_ACCOUNT_A:

gcloud iam service-accounts keys create \
./some-key.json \
--iam-account="${SERVICE_ACCOUNT_A}" \
--key-file-type="json"

#=>

created key [$KEY_ID] of type [json] as [./some-key.json] for [$SERVICE_ACCOUNT_A]

activate the service account:

gcloud auth activate-service-account \
"${SERVICE_ACCOUNT_A}" \
--key-file=./some-key.json

#=>

Activated service account credentials for: [$SERVICE_ACCOUNT_A]

confirm it's active:

cloud auth list
                        Credentialed Accounts
ACTIVE  ACCOUNT
        . . .
*       $SERVICE_ACCOUNT_A
        your@account.user
        . . .

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

and attempt to interact with $GKE_CLUSTER_B:

gcloud container clusters list --project=$PROJECT_B

#=>

ERROR: (gcloud.container.clusters.list) ResponseError: code=403, message=Kubernetes Engine API has not
been used in project $PROJECT_A_ID before or it is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/container.googleapis.com/overview?project=$PROJECT_A_ID
then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our
systems and retry.

where $PROJECT_A_ID is a numeric id of the form: xxxxxxxxxxxxx.

Visit the address returned along with the 403 above (here) and enable the Kubernetes Engine API. $SERVICE_ACCOUNT_A should now be able to interact with GKE clusters within $PROJECT_B:

gcloud container clusters list \
--project=$PROJECT_B \
--format='value(name)

#=>

. . .
some_cluster
. . .

including $GKE_CLUSTER_B.

Service Account Unable to Interact with GKE Cluster in Another Project

1 Answers