Spotify spark-bigquery connector issue while using BigQuerySelect method from Dataproc cluster

1
votes

I am new to BigQuery GCP and to access BigQuery data we are using Spotify spark-bigquery connector as provided here.

We are able to use sqlContext.bigQueryTable("project_id:dataset.table") and its working.

When we are using sqlContext.bigQuerySelect("SELECT * FROM [project_id:dataset.table]") it is giving error:

The user [email protected] does not have permission to query table.

We have done necessary settings w.r.t json file and location. But don't have any clue about from where it is taking this user account details.

Please provide help regarding its cause and how to fix it in code.

1
google-cloud-platformgoogle-bigqueryspotifygoogle-cloud-dataproc
We have resolved it by creating json file and then setting export GOOGLE_APPLICATION_CREDENTIALS=<path_to_service_account_file> on dataproc cluster because we are using it on dataproc cluster. But don't know cause of this way of working of API. – Sarang Shinde

1 Answers

0
votes

This error indicates that the service account you are using ([email protected]) doesn’t have enough IAM permissions. You should go to your IAM settings and make sure it has at least BigQuery Data Viewer permissions.