kubernetes external secrets on GKE , Permission error

Question

I install kubernetes external secrets with helm, on GKE.

• GKE: 1.16.15-gke.6000 on asia-northeast1
• helm app version 6.2.0
• using Workload Identity as document written

For workload identity,the service account I bind as below command (my-secrets-sa@$PROJECT.iam.gserviceaccount.com) has SecretManager.admin role, which seems necessary for using google secrets manager

gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:$CLUSTER_PROJECT.svc.id.goog[$SECRETS_NAMESPACE/kubernetes-external-secrets]" my-secrets-sa@$PROJECT.iam.gserviceaccount.com

Workload identity looks set correctly, because checking service account in pod on GKE shows correct serviceaccount

https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_workload_identity_on_a_new_cluster

create a pod in cluster and check auth in it. it shows my-secrets-sa@$PROJECT.iam.gserviceaccount.com

$ kubectl run -it --image google/cloud-sdk:slim --serviceaccount ksa-name --namespace k8s-namespace workload-identity-test

$ gcloud auth list

But even if creating externalsecret, externalsecret shows error

ERROR, 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/project-id/secrets/my-gsm-secret-name/versions/latest' (or it may not exist).

secret my-gsm-secret-name itself exist in secretmanager, so it should not "not exist". Also permission must be correctly set by workload identity.

it's the externalsecret I defined.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: gcp-secrets-manager-example    # name of the k8s external secret and the k8s secret
spec:
  backendType: gcpSecretsManager
  projectId: my-gsm-secret-project
  data:
    - key: my-gsm-secret-name     # name of the GCP secret
      name: my-kubernetes-secret-name   # key name in the k8s secret
      version: latest    # version of the GCP secret
      property: value      # name of the field in the GCP secret

Has anyone had similar problem before ? Thank you

whole command

1. create a cluster with workload-pool.

$ gcloud container clusters create cluster --region asia-northeast1 --node-locations asia-northeast1-a --num-nodes 1 --preemptible --workload-pool=my-project.svc.id.goog

2. create kubernetes service account.

$ kubectl create serviceaccount --namespace default ksa

3. binding kubernetes service account & service account

$ gcloud iam service-accounts add-iam-policy-binding
--role roles/iam.workloadIdentityUser
--member "serviceAccount:my-project.svc.id.goog[default/ksa]"
[email protected]`

4. add annotation

$ kubectl annotate serviceaccount
--namespace default
ksa
iam.gke.io/gcp-service-account=[e-mail]

5. install with helm

$ helm install my-release external-secrets/kubernetes-external-secrets

6. create external secret

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: gcp-secrets-manager-example    # name of the k8s external secret and the k8s secret
spec:
  backendType: gcpSecretsManager
  projectId: my-gsm-secret-project
  data:
    - key: my-gsm-secret-name     # name of the GCP secret
      name: my-kubernetes-secret-name   # key name in the k8s secret
      version: latest    # version of the GCP secret
      property: value      # name of the field in the GCP secret

$ kubectl apply -f excternal-secret.yaml

Answer 1

I noticed that I had used different kubernetes service account.

When installing helm, new kubernetes service account my-release-kubernetes-external-secrets was created, and service/pods must be working on this service account.

So I should bind my-release-kubernetes-external-secrets & google service account.

Now, it works well.

Thank you @matt_j @norbjd