Dataflow appears to be stuck 3

0
votes

I got this error on Dataflow log:

(11b55950300ec4ce): Workflow failed. Causes: (11b55950300ec459): The Dataflow appears to be stuck. Please reach out to the Dataflow team at http://stackoverflow.com/questions/tagged/google-cloud-dataflow.

The JobID is: 2017-11-13_20_58_16-5812097262489045871

I tried the exact same code and it is working on other project...

Any help is appreciated,

Yosua

1
google-cloud-dataflow
I tried the WordCount code and it got the same error on the same project. Error id: (efbac2f6c2ec13f0) JobID: 2017-11-14_00_42_11-4211727424575330420 - Yosua
It looks like Dataflow failed while trying to start GCE VMs to run the job. Is it possible that you renamed the default network ? If so you have to specify the network using pipeline option github.com/apache/beam/blob/master/runners/… - chamikara
@chamikara Yeah, it seems to be the network problem. It works after I specify the network in the pipeline option. Thanks a lot! - Yosua

1 Answers

2
votes

It turns the dataflow cannot spawn workers because of mismatch between the default network's region (in asia-southeast1 in my case) and the zone parameters (default: us-central1-f).

So to solve this issue I need to explicitly set the network, subnet, and zone i the pipeline parameter.

Note: I found this unintuitive but it is okay to have region as us-central1 and zone outside us-central1 since the region parameter is where we deploy the jobs and zone is where we spawn the worker.

Reference: https://cloud.google.com/dataflow/pipelines/specifying-exec-params