Google Dataflow - Invalid Regional Endpoint - Impossible set region on template from nodejs client

Question

I have a pipeline stored as a template. I'm using the node.js client to run this pipeline from a cloud function. Everything works fine, but when I need to run this template from different regions I get errors.

According to the documentation, I can set it through the location parameter in the payload

{
  projectId: 123,
  resource: {
    location: "europe-west1",
    jobName: `xxx`,
    gcsPath: 'gs://xxx'
  }
}

That gives me the following error:

The workflow could not be created, since it was sent to an invalid regional endpoint (europe-west1). 
Please resubmit to a valid Cloud Dataflow regional endpoint.

I get the same error if I move the location parameter out of resource node, such as:

{
  projectId: 123,
  location: "europe-west1",
  resource: {
    jobName: `xxx`,
    gcsPath: 'gs://xxx'
  }
}

If I set the zone in the environment and remove the location such as:

{
  projectId: 123,
  resource: {
    jobName: `xxx`,
    gcsPath: 'gs://xxx',
    environment: {
        zone: "europe-west1-b"
    }
   }
}

I do not get any errors anymore, but dataflow UI tells me the job is running in us-east1

How can I run this template and providind the region / zone I

Does it work if you specify us-central1 using location too? Are you using a deprecated SDK or Shuffle service that do not support all regional endpoints? — Guillem Xercavins
Yes it does if I specify us-central1. I'm using the latest googleapis version — benjamin.d

Unknown Unknown · Accepted Answer · 2018-03-19T08:50:32

As explained here, there are actually two endpoints:

dataflow.projects.locations.templates.launch (API Explorer)
dataflow.projects.templates.launch (API Explorer)

For Dataflow regional endpoints to work, the first one must be used (dataflow.projects.locations.templates.launch). This way, the location parameter in the request will be accepted. Code snippet:

var dataflow = google.dataflow({
    version: "v1b3",
    auth: authClient
});

var opts = {
    projectId: project,
    location: "europe-west1",
    gcsPath: "gs://path/to/template",
    resource: {
        parameters: launchParams,
        environment: env
    }
};
dataflow.projects.locations.templates.launch(opts, (err, result) => {
    if (err) {
        throw err;
    }
    res.send(result.data);
});

Google Dataflow - Invalid Regional Endpoint - Impossible set region on template from nodejs client

2 Answers