I am trying to read in a bigquery dataset in dataflow. It cannot find the bigquery dataset/table that I specify.
job_name is preprocess-ga360-190523-130005
my datalab vm, gcs bucket, and bigquery dataset are all location in europe-west2.
For some reason, it is searching for the dataset in location 'US'
modules versions are apache-beam 2.5.0,google-cloud-dataflow 2.0.0, google-cloud-bigquery 0.25.0
Searched documentation and couldn't come up for an answer as to why this is happening.
OUTPUT_DIR = "gs://some-bucket/some-folder/"
#dictionary of pipeline options
options = {
"staging_location": "gs://some-bucket/some-folder/stage/"
"temp_location": "gs://some-bucket/some-folder/tmp/"
"job_name": job_name,
"project": PROJECT,
"runner": "DirectRunner",
"location":'europe-west2',
"region":'europe-west2',
}
#instantiate PipelineOptions object using options dictionary
opts = beam.pipeline.PipelineOptions(flags = [], **options)
#instantantiate Pipeline object using PipelineOptions
with beam.Pipeline(options=opts) as
outfile = "gs://some-bucket/some-folder/train.csv"
(
p | "read_train" >> beam.io.Read(beam.io.BigQuerySource(query =
my_query, use_standard_sql = True))
| "tocsv_train" >> beam.Map(to_csv)
| "write_train" >> beam.io.Write(beam.io.WriteToText(outfile))
)
print("Done")
Response:
HttpError: HttpError accessing https://www.googleapis.com/bigquery/v2/projects/projects/queries/querystring: response: <{'status': '404', 'content-length': '342', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Thu, 23 May 2019 13:00:08 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}>, content <{ "error": { "code": 404, "message": "Not found: Dataset my_dataset:views was not found in location US", "errors": [ { "message": "Not found: Dataset my_dataset:views was not found in location US", "domain": "global", "reason": "notFound" } ], "status": "NOT_FOUND" } }
