I have a big query instance which I want to connect to Airflow. I do so by adding the following connection variable:
{
"conn_id": "bigquery_default",
"conn_type": "gcp",
"extra": {
"extra__google_cloud_platform__key_path" : "/path-to-credentials.json",
"extra__google_cloud_platform__scope" : "https://www.googleapis.com/auth/bigquery",
"extra__google_cloud_platform__project" : "project-name-here"
}
}
This gives me an error when attempting to use a GCS to BQ operator:
HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/project-name-here/jobs?alt=json returned "Required parameter is missing".
If I change the scope from https://www.googleapis.com/auth/bigquery to https://www.googleapis.com/auth/bigquery.insertdata, I get the following error:
HttpError 403 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/project_name_here/datasets/DATASET_NAME_HERE/tables/TABLE_NAME_HERE?alt=json returned "Request had insufficient authentication scopes."
When I add https://www.googleapis.com/auth/cloud-platform scope in addition to the bigquery scope, I no longer get the "required parameter missing" error, but instead get a token error.
I suspect something is not lining up with the service account which was created in regard to the big query connection entry I'm putting into airflow. The earlier error regarding insufficient scopes is telling me maybe there are permission issues? But why does it keep saying "missing parameter"? Why do I need to add multiple scopes? I have limited experience with GCP, but when I have created service accounts, it made no mention of assigning a "scope" to a service account.
The credentials and connection variable for the GCP instance which hosts the GCS bucket is correct as I have been able to write data from an external source to the GCS bucket successfully.
