What Service Account Does GKE Use to Access GCR

4
votes

One of the Google Kubernetes Engine (GKE) clusters ($GKE_CLUSTER) within a Google Cloud Platform (GCP) project ($GCP_PROJECT) seems to be unable to pull Docker Images from Google Container Registry (GCR):

kubectl config current-context

#=>

$GKE_CLUSTER

and:

kubectl get pods --namespace=$NAMESPACE

#=>

NAME        READY   STATUS             RESTARTS   AGE 
. . .       . . .   . . .              . . .      . . .
$SOME_POD   0/1     ImagePullBackOff   0          1m
. . .       . . .   . . .              . . .      . . .

and:

kubectl get events \
--field-selector involvedObject.name=$SOME_POD \
--namespace=$NAMESPACE

#=>

LAST SEEN   TYPE     REASON    OBJECT          MESSAGE
. . .       . . .    . . .     . . .           . . .
1m          Normal   BackOff   pod/$SOME_POD   Back-off pulling image "$SOME_IMAGE"
. . .       . . .    . . .     . . .           . . .

Where:

  • $SOME_POD is of the form: deployment-replicaSet-pod
  • $SOME_IMAGE is of the form: us.gcr.io/$GCP_PROJECT/name:tag

Listing only service accounts for $GCP_PROJECT gives:

gcloud projects get-iam-policy $GCP_PROJECT \
--filter="serviceAccount" \
--flatten="bindings[].members" \
--format="value(bindings.members.split(':').slice(1:).flatten())"

#=>

[email protected]
[email protected] 
[email protected]  
[email protected]
service-XXXXXXXXXXXX@container-engine-robot.iam.gserviceaccount.com
[email protected]  
[email protected]    
[email protected]
[email protected]

Which of these service accounts is GKE using to access Docker Images hosted by GCR?

3
google-cloud-platformgoogle-compute-enginegoogle-kubernetes-enginegoogle-container-registrygoogle-iam
Compute Engine default service account is used by default when you create a new cluster. Could you provide more information about the error you're facing? – Alexandre
Unfortunately I don't have the specific error because by handing around enough permissions it was solved. The error was that a repository did not exist or not permissions to it. I verified it existed – Mugen

3 Answers

6
votes

Each GKE node has an IAM Service Account associated with it. By default, nodes are given the Compute Engine default service account, which you can find by navigating to the IAM section of the Cloud Console.

If you are using the non default Compute Engine service account you probably will need to grant the service account roles/storage.objectViewer role in the project. Check this link for more information.

1
votes

For GKE you will have an account ending like this "container-engine-robot.iam.gserviceaccount.com" if you have any issue (that could be related with some changes over the API) you can remove the default accounts for GCE and GKE (Under "Name" you will see the details and which belongs to each resource) and re enable the service using the gcloud command to re-create your default service accounts, if not just assign the "Editor" role and try again.

1
votes

Google Kubernetes Engine (GKE) doesn’t access Google Container Registry (GCR) directly: one or more node pools associated with the GKE cluster push and pull Docker images through the Container Registry API, authenticated and authorized through service accounts.

If your GKE cluster has a single node pool:

gcloud container clusters describe $GKE_CLUSTER_NAME \
--format='value(nodeConfig.serviceAccount)' \
--zone=$GKE_CLUSTER_ZONE

#=>

default

If your GKE cluster has multiple node pools:

gcloud container clusters describe $GKE_CLUSTER_NAME \
--zone=$GKE_CLUSTER_ZONE \
--format='value[delimiter="\n"](nodePools[].config.serviceAccount)'

#=>

default
not-default@$GCP_PROJECT_NAME.iam.gserviceaccount.com
. . .

default here is the “Compute Engine default service account”: [email protected].

Note: default serv. acct. differ from serv. acct. that you are able to create: beginning with a number and ending in a domain other than: @$GCP_PROJECT_NAME.iam.gserviceaccount.com.

In order for a serv. acct. to be able to pull Docker images from GCR it needs to have at least one associated role with the following associated permissions:

  • storage.objects.get
  • storage.objects.list

You might encounter:

WARNING: You do not appear to have access to project [$GCP_PROJECT_NAME] or it does not exist.

if the serv. acct. does not have at least one associated role with the following associated permissions:

  • resourcemanager.projects.get
  • resourcemanager.projects.list

The curated role that contains all four of the above permissions is the roles/storage.objectViewer role.

If you ever want to store Docker images in another GCP project, you just need to grant the serv. acct. used by the node pool(s) within your GKE cluster the roles/storage.objectViewer role in the project your would like to store the Docker images (more on that here).

Note: you may wish to use GCP Artifact Registry (AR) going forward. More on transitioning from GCR to AR here.