Network default is not accessible to Dataflow Service account

10
votes

Having issues starting a Dataflow job(2018-07-16_04_25_02-6605099454046602382) in a project without a local VPC Network when I get this error 

Workflow failed. Causes: Network default is not accessible to Dataflow
Service account

There is a shared VPC connected to the project with a networked called default with a subnet default in us-central1 – however the service account used to run dataflow job don't seam to have access to it. I have given the dataflow-service-producer service account Compute Network User, without any noticeable effect. Any ideas on how I can processed?

2
google-cloud-platformgoogle-cloud-dataflow

2 Answers

7
votes

The usage of subnetworks in Cloud Dataflow requires to specify the subnetwork parameter when running the pipeline; However, in the case of subnetwork that are located in a Shared VPC network, it is required to use the complete URL based on the following format, as you well mentioned.

https://www.googleapis.com/compute/v1/projects/<HOST_PROJECT>/regions/<REGION>/subnetworks/<SUBNETWORK>

Additionally, in this cases is recommended to verify that you are adding the project's Dataflow service account into the Shared VPC's project IAM table and give it the "Compute Network User" role permission in order to ensure that the service has the required access scope.

Finally, it is seems that the Subnetwork parameter official Google's documentation is alraedy available with detailed information about this matter.

2
votes

Using the --subnetwork option with the following (undocumented) fully qualified subnetwork format made the Dataflow job run. Where {PROJECT} is the name of the project hosting the shared VPC and {REGION} matches the region you run your dataflow job in. 

--subnetwork=https://www.googleapis.com/compute/alpha/projects/{PROJECT}/regions/{REGION}/subnetworks/{SUBNETWORK}