Request had insufficient authentication scopes

Question

I am trying to run pyspark application using gcloud on dataproc master-node . I get "Request had insufficient authentication scopes"

# gcloud dataproc jobs submit pyspark --cluster xxxxx test.py

gcloud.dataproc.jobs.submit.pyspark) You do not have permission 
to access        cluster [xxxxxx] (or it may not exist): 
Request had  insufficient authentication scopes

I can run same application through Jobs GUI. I dont have the link to doc rightnow but it says if this is been run on Compute VM no separate credential is required which seem to be in line with when I run the same application using GUI. Any help ?

Answer 1

When running from a Dataproc node, you'll be acting on behalf of a service account attached to the VM. Usually you'll be using a default compute engine service account, but this can also be specified with the Dataproc service account configuration. Alongside a service account, there's also a list of scopes, which limit which GCP services that service account is allowed to access from that VM. By default, there is BigQuery, GCS, logging, and some other small scopes, but not a general administrative scope for doing things like creating other VMs or Dataproc clusters.

To grant the necessary scope, you have to add --scopes when first creating your cluster:

gcloud dataproc clusters create --scopes cloud-platform ...