How to change default network with Google-Provided Dataflow Template

Question

I am trying to setup a Dataflow job using the google provided template PubSub to BigQuery. However I am getting this error on start up:

Message: The resource 'projects/my-project/global/networks/default' was not found

I think the google provided template is hardcoded to use the default network. The error goes away if I create a default network in automatic mode. But we can't have a default network in production.

The document here mentions a network parameter. I tried adding an additional parameter called network from GCP console UI passing in our custom network name. But I am getting this error:

The template parameters are invalid.

Is there any way I can tell the Google provided Dataflow template to use my custom network(created in manual mode) instead of the default? What are my options here?

Appreciate all the help!

Answer 1

This is not currently supported for Dataflow pipelines created from a template. For now, you can either run the template in the default VPC network, or submit a Dataflow pipeline using the Java or Python SDK and specify the network pipeline option.

Answer 2

You can use gcloud beta command gcloud beta dataflow jobs run as explained here in gcloud beta dataflow jobs run. It supports more parameters like [--network=NETWORK] and [--subnetwork=SUBNETWORK] which are useful for your usecase.