Unable to run Dataflow job due to region + zone limitation

Question

We have our data and our network configured in northamerica-northeast region.

We want to run the data flow job to process our input file and load in BigQuery table. Our storage and BigQuery is also configured in same region northamerica-northeast1.

However, when we run the job : we get the following error -

The workflow could not be created, since it was sent to an invalid or unreleased region. Please resubmit with a valid region.",

We are passing the following arguments to our data flow job: --region northamerica-northeast1 --zone northamerica-northeast1-a

Now as per below KB - https://cloud.google.com/dataflow/docs/concepts/regional-endpoints Dataflow does not have a regional-endpoint in northamerica-northeast1. However we can override the zone.

Any assistance on how we can do the same. How can we run the job then in northamerica-northeast1.

Answer 1

You can look at this table https://cloud.google.com/dataflow/docs/concepts/regional-endpoints#commonscenarios. For the scenario that you have mentioned below setup has to be done

I need worker processing to occur in a specific region that does not have a regional endpoint.  

Specify both --region and --zone.
Use --region to specify the supported regional endpoint that is closest to the zone where the worker processing must occur. Use --zone to specify a zone within the desired region where worker processing must occur.