How to set Job Region for Template Dataflow Job not only Pipeline Region?

0
votes

I would like to start GCP Dataflow from Custom Dataflow Template, with Region set to Europe. As result I'm getting region equal to europe-west1 in Pipeline Options and my VM is running in Europe too, but Job option's region is set to us-central1 (in Job summary part - region where metadata is stored and handled for this job).

When I uploaded my template I put in options --region=europe-west1. When I start my dataflow from template I also set region to europe-west1. I tried to set zones as well and set region to Europe wherever it is possible, but I still was getting the same result. Documentation on setting Job region - regional endpoints is only mentioning this one parameter (region).

Am I missing something? How can I set Job region to Europe too?

UPDATE: It worked from command line with --region=europe-west1, but I need to start my Dataflow from Airflow.

2
google-cloud-platformgoogle-cloud-dataflow

2 Answers

1
votes

It seems that the custom template UI is missing the option to select a region. I followed up with the Dataflow team to address this. Thank you for pointing this out.

Instead you might be able to use the command line tool to launch the template job. Please give this a try

gcloud beta dataflow jobs run myJob --gcs-location=<gcs_template_location> --region=europe-west1

Usage: gcloud beta dataflow jobs run JOB_NAME --gcs-location=GCS_LOCATION 
[optional flags]
optional flags may be --help | --max-workers | --network | --num-workers |
--parameters | --region | --service-account-email |
--staging-location | --subnetwork |
--worker-machine-type | --zone
0
votes

Older version of Airflow was pointing to wrong endpoints, the one with no region parameter.

I could see GCP UI addressed this issue too and it's now possible to start Dataflow Template from UI with proper region (regional endpoints).