Google Cloud Java SDK with Workload Identity?

Question

Trying to figure out how to authenticate with the storage API from within a GKE cluster.

Code:

Storage storage = StorageOptions.newBuilder()
  .setCredentials(ServiceAccountCredentials.getApplicationDefault())
  .setProjectId(gcpProjectId)
  .build().getService();

getApplicationDefault() is documented to use these means to authenticate with the API:

1. Credentials file pointed to by the {@code GOOGLE_APPLICATION_CREDENTIALS} environment variable
2. Credentials provided by the Google Cloud SDK {@code gcloud auth application-default login} command
3. Google App Engine built-in credentials
4. Google Cloud Shell built-in credentials
5. Google Compute Engine built-in credentials

The application is using the GCP workload identity feature, so the application (in-cluster) service account is annotated with:

serviceAccount.annotations.iam.gke.io/gcp-service-account: [email protected]

Now the call to the storage account fails with the following error:

{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Primary: /namespaces/my-project.svc.id.goog with additional claims does not have storage.objects.create access to the Google Cloud Storage object.",
    "reason" : "forbidden"
  } ],
  "message" : "Primary: /namespaces/my-project.svc.id.goog with additional claims does not have storage.objects.create access to the Google Cloud Storage object."
}

This makes me think that the workload identity is not working correctly. I am expecting to receive an error message for my annotated service account and not the default one.

Is there anything else I should have been doing?

Answer 1

The answer, in part, aside from the annotation syntax, is that, just like me, you probably didn't look closely enough at this part in the documentation:

    gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Notice the PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME] piece. It's something they give no examples on as far as syntax but it looks like this in my terraform.

resource "google_project_iam_member" "app-binding-2" {
  role   = "roles/iam.workloadIdentityUser"
  member = "serviceAccount:${local.ws_vars["project-id"]}.svc.id.goog[mynamespace/myk8ssaname]"
}

Weirdly, I didn't know you could bind an IAM policy to a k8s service account, even more weirdly you can bind this in the terraform even if the namespace doesn't exist, much less the service account. So you can run this first before deployments.

I truly wish Google would provide better documentation and support, this took me several hours to figure out.

Answer 2

The annotation is wrong. Instead of:

serviceAccount.annotations.iam.gke.io/gcp-service-account: [email protected]

it must be

iam.gke.io/gcp-service-account: [email protected]

Now the error message also shows that the app is using a workload identity:

java.io.IOException: Unexpected Error code 403 trying to get security access token from Compute Engine metadata for the default service account: Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation:
    https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#creating_a_relationship_between_ksas_and_gsas