Not able to access Datastore Resources from GKE

Question

I have deployed an app in GKE which makes a backend call to datastore to perform crud operations.Added cloud datastore owner role to the Service account on which gke is hosted.

when i request any of the endpoint which makes call to the backend datastore i am getting below excpetion:

com.google.cloud.datastore.DatastoreException: Unauthenticated. at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:123) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:173) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:416) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:413) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[gax-1.58.2.jar:1.58.2] at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[google-cloud-core-1.93.9.jar:1.93.9] at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[google-cloud-core-1.93.9.jar:1.93.9] at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:412) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.loadResults(DatastoreImpl.java:387) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.<init>(DatastoreImpl.java:383) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:373) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:336) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreHelper.fetch(DatastoreHelper.java:73) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at com.google.cloud.datastore.DatastoreImpl.fetch(DatastoreImpl.java:353) ~[google-cloud-datastore-1.105.0.jar:1.105.0] at org.springframework.cloud.gcp.data.datastore.core.DatastoreTemplate.findAllById(DatastoreTemplate.java:249)

One thing i noticed is that when the application start it uses default compute engine credentials to authenticate

I am not sure on how to fix this , searched for few solutions and came to know that we need to add GOOGLE_APPLICATION_CREDENTIALS env variable in deployment.yaml

spec:
 containers:
 - name: app
   image: eu.gcr.io/google_project_id/springapplication:v1
   volumeMounts:
   - name: google-cloud-key
     mountPath: /var/secrets/google
   env:
   - name: GOOGLE_APPLICATION_CREDENTIALS
     value: /var/secrets/google/credentials.json
   ports:
   - name: http-server
     containerPort: 8080
 volumes:
 - name: google-cloud-key
   secret:
     secretName: app-key

Is this the right way, since the credentials file is a confidential one where we store it ideally(I am using Jenkins to deploy apps)

Someone please guide me on this.....

guillaume blaquiere guillaume blaquiere · Accepted Answer · 2021-01-15T20:44:11

It's a problem at node level permission. When you create your cluster, and your node pool, you can choose the security of your node. And you can explicitly allow or deny the access to some API, such as Datastore. And by default Datastore is disabled.

You can check this in the Compute Engine that compose your GKE cluster:

If you don't want to delete your Cluster, you need to create a new node pool, to migrate your pod and to remove the old node pool.

Side remark. Don't you Service account key files. I recommend you to use workload identity for a better security

Not able to access Datastore Resources from GKE

1 Answers