BigQuery GCP Python Integration

0
votes

I am trying to write all my scripts in Python instead of BigQuery. I set my active project using 'glcoud config set project' but I still get this ERROR 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/analytics-supplychain-thd/jobs: Caller does not have required permission to use project analytics-supplychain-thd. Grant the caller the Owner or Editor role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=analytics-supplychain-thd and then retry (propagation of new permission may take a few minutes).

How do I fix this?

1
pythongoogle-cloud-platformgoogle-bigquery
Can you please add info regarding exactly what are you trying to do? how are you trying to do it? – Alonme
what command to you run? What is the full error trace? – guillaume blaquiere
Read this link and then edit your question to be suitable for Stack Overflow. stackoverflow.com/help/minimal-reproducible-example – John Hanley

1 Answers

1
votes

I suspect you are picking up the wrong "key".json, at least in terms of permissions for one of the operation you are trying to perform. The key currently defined [1] in GOOGLE_APPLICATION_CREDENTIALS seems not have right permission. A list of roles you should grant to the Service Account can be find here [2], anyway from your error you would need at least a primitive role as Owner or Editor. The latter depends on your needs and targets (operation you perform through such script).

You should pick up the right role for your operation and associating it to the Service Account you want to use, defining therefore an identity for it through the IAM portal UI, also doable also through the CLI or API calls.

After that be sure the client you are using is logged in with the correct service account (correct json key path).

Particularly, I used the code you gave me to test and I have been able to load the data:

import pandas_gbq 
import google.oauth2.service_account as service_account
# TODO: Set project_id to your Google Cloud Platform project ID 
project_id = "xxx-xxxx-xxxxx" 
sql = """SELECT * FROM xxx-xxxx-xxxxx.fourth_dataset.2test LIMIT 100""" 
credentials = service_account.Credentials.from_service_account_file('/home/myself/key.json')
df = pandas_gbq.read_gbq(sql, project_id=project_id, dialect="standard", credentials=credentials)

This Hope this helps!!

[1] https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable

[2] https://cloud.google.com/iam/docs/understanding-roles#primitive_roles