Templated Dataflow hangs over an hour at loading and quits with an error

1
votes

When translating a Dataflow from command line execution to templatable execution, I encountered the following problem.

The Dataflow starts but hangs at the loading state just after the following log message:

(71df0de383b642bd): Starting 1 workers in europe-west2-a...

After an hour, it seems like a time-out triggers and the Dataflow stops with the following log message:

(84598aaa4185b9a0): Workflow failed. Causes: (84598aaa4185b571): The Dataflow appears to be stuck. Please reach out to the Dataflow team at http://stackoverflow.com/questions/tagged/google-cloud-dataflow.

I followed the translation guide from: https://cloud.google.com/dataflow/docs/templates/creating-templates. I used the "RuntimeValueProvider" to catch all my arguments which I normally inserted via command line.

Can I get any help on this?

1
pythongoogle-cloud-dataflow
Sure. Can you provide a Job ID? – Pablo
Hello Pablo, please see Job ID 2017-11-02_10_31_06-3577732489891572038 and 2017-11-02_08_29_42-10298347787797825179 – Martin van Dam
Did you provide a zone and a region flag for this? And if so, which ones did you provide? – Pablo
I believe that your problem is that in the template you set region,zone = europe-west2-a, but submit the job to region=europe-west1. Please don't set the region for your template, or set it to the appropriate europe-west1. – Pablo
Dear Pablo, Oke that's fair. But what about 2017-11-06_11_05_25-4140564809200439667 ? That template is not created with a predefined region. – Martin van Dam

1 Answers

2
votes

It appears to be that at creation time, I had only set a --zone parameter without a --region parameter. This results in a template file having the --region option set to it's default ("us-central1" at this moment) and leaving the --zone option to whatever I had set it. In my case, this was "europe-west1-b".

So having a template with --region us-central1 and --zone europe-west1-b is failing. Kind of logically in the end. But... There was no feedback on this. Because if you don't set the --region flag at template creation, neither de default value is rendered in the information pane. So it might come handy if these values can be rendered into the information pane on the right. Or even highlight mismatches that may cause such a failure.

Now, in the end, I create my templates with --region europe-west1 and --zone europe-west1-b to fix this. (I thought it also works without the supplementing --zone option, but I just pick one already)

Thanks a lot Pablo, for pointing me to the regions. My templates work now.