How to solve insufficient authentication scopes when use Pubsub on GCP

Question

I'm trying to build 2 microservices (in Java Spring Boot) to communicate with each other using GCP Pub/Sub.

First, I tested the programs(in Eclipse) working as epxected in my local laptop(http://localhost), i.e. one microservice published the message and the other received it successfully using the Topic/Subscriber created in GCP (as well as the credential private key: mypubsub.json).

Then, I deployed the same programs to run GCP, and got following errors:

- 2020-03-21 15:53:16.831 WARN 1 --- [bsub-publisher2] o.s.c.g.p.c.p.PubSubPublisherTemplate : Publishing to json-payload-sample-topic topic failed
- com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Request had insufficient authentication scopes. at com.google.api.gax.rpc.ApiExceptionFactory

What I did to deploy the programs(in container) to run on GCP/Kubernetes Engine:

Login the Cloud Shell after switch to my project for the Pubsub testing
Git clone my programs which being tested in Eclipse
Move the mypubsub.json file to under /home/my_user_id
export GOOGLE_APPLICATION_CREDENTIALS="/home/my_user_id/mp6key.json"
Run 'mvn clean package' to build the miscroservice programs
Run 'docker build' to create the image files
Run 'docker push' to push the image files to gcr.io repo
Run 'kubectl create' to create the deployments and expose the services

Once the 2 microservices deployed and exposed, when I tried to access them in browser, the one to publish a message worked fine to retrieve data from database and processed the data, then failed with the above errors when trying to access the GCP Pubsub API to publish the message

Could anyone provide a hint for what to check to solve the issue?

Edit your question to clarify. You mention Cloud Shell, Compute Engine and Kubernetes. What are you using and for which part? Which roles have you assigned to the service account that the container is using? stackoverflow.com/help/minimal-reproducible-example — John Hanley
It's nothing related to Compute Engine. Basically it's that I used the Cloud Shell as the tool to deploy my programs as Container to run on GCP (i.e. using the Google Kubernetes Engine to run a container of my application instead of a Cloud Run). I assiend the service account as the Pub/Sub Admin of the Topic for which my program trying to publish a message. — Roy Hu
Then why do you say "I also tried to stop the VM instance then update the service account`? Your question needs improvement in focus and clarity. As written I can not even try to reproduce what you have done, nor figure out what your real problem is. Read my first comment again and then read this link: stackoverflow.com/help/how-to-ask — John Hanley
@John: It's my understanding issue: I thought I must create a VM instance even though I run my application by Contrainer(GKE) on GCP, thus I googled a sotution for my issue that updating the service account to 'Allow full access to all Cloud APIs' after stopping the VM instance. With your last comment, I realized my project nothing related to Compute Engine thus may not need to creat a VM instance at all. I just tried to stop the VM instance and re-run my services on GCP, but still the same issue... PS: I removed the VM related part from my question so that to not confuse others. — Roy Hu

Roy Hu Roy Hu · Accepted Answer · 2020-03-22T17:38:03

The issue has been resolved by following the guide:

https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform

Briefly the solution is to add following lines in the deployment.yaml to load the credential key:

        - name: google-cloud-key
          secret:
            secretName: pubsub-key
        containers:
        - name: my_container
          image: gcr.io/my_image_file
          volumeMounts:
          - name: google-cloud-key
            mountPath: /var/secrets/google
          env:
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /var/secrets/google/key.json

How to solve insufficient authentication scopes when use Pubsub on GCP

1 Answers