How do I present a custom GCP service account to kubernetes workloads?

Question

I'm configuring a highly available kubernetes cluster using GKE and terraform. Multiple teams will be running multiple deployments on the cluster and I anticipate most deployments will be in a custom namespace, mainly for isolation reasons.

One of our open questions is how to manage to manage GCP service accounts on the cluster.

I can create the cluster with a custom GCP service account, and adjust the permissions so it can pull images from GCR, log to stackdriver, etc. I think this custom service account will be used by the GKE nodes, instead of the default compute engine service account. Please correct me if I'm wrong on this front!

Each deployment needs to access a different set of GCP resources (cloud storage, data store, cloud sql, etc) and I'd like each deployment to have it's own GCP service account so we can control permissions. I'd also like running pods to have no access to the GCP service account that's available to the node running the pods.

Is that possible?

I've considered some options, but I'm not confident on the feasibility or desirability:

A GCP Service account for a deployment could be added to the cluster as a kubernetes secret, deployments could mount it as a file, and set GOOGLE_DEFAULT_CREDENTAILS to point to it
Maybe access to the metadata API for the instance can be denied to pods, or can the service account returned by the metadata API be changed?
Maybe there's a GKE (or kubernetes) native way to control the service account presented to pods?

Artem Golenyaev Artem Golenyaev · Accepted Answer · 2018-06-08T15:55:56

You are on the right track. GCP service accounts can be used in GKE for PODs to assign permissions to GCP resources.

Create an account:

cloud iam service-accounts create ${SERVICE_ACCOUNT_NAME}

Add IAM permissions to the service account:

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role='roles/${ROLE_ID}'

Generate a JSON file for the service account:

gcloud iam service-accounts keys create \
  --iam-account "${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
  service-account.json

Create a secret with that JSON:

kubectl create secret generic echo --from-file service-account.json

Create a deployment for your application using that secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      name: echo
    spec:
      containers:
        - name: echo
          image: "gcr.io/hightowerlabs/echo"
          env:
            - name: "GOOGLE_APPLICATION_CREDENTIALS"
              value: "/var/run/secret/cloud.google.com/service-account.json"
            - name: "PROJECT_ID"
              valueFrom:
                configMapKeyRef:
                  name: echo
                  key: project-id
            - name: "TOPIC"
              value: "echo"
          volumeMounts:
            - name: "service-account"
              mountPath: "/var/run/secret/cloud.google.com"
      volumes:
        - name: "service-account"
          secret:
            secretName: "echo"

If you want to use various permissions for separate deployments, you need to create some GCP service accounts with different permissions, generate JSON tokens for them, and assign them to the deployments according to your plans. PODs will have access according to mounted service accounts, not to service the account assigned to the node.

For more information, you can look through the links:

How do I present a custom GCP service account to kubernetes workloads?

1 Answers