Cloud Dataflow - Worker pool teardown error

1
votes

Our pipeline, which has been running fine for a few months, threw the following error today. It looks like the worker pool was unable to be torn down after the pipeline finished:

Aug 31, 2015, 4:35:55 AM (95577a18aba47708): Workflow failed. Causes: (df2004d5f7046091): Step teardown_resource_global_gce_worker_pool2: Resource worker_pool_resource failed to shut down

Job_ID: 2015-08-30_11_01_50-8584391373463411570

Is this a known issue?

1
google-cloud-dataflow
Sorry for the trouble. We'll investigate. Did your VMs get shut down? – Frances
They did, but I've no idea how long it took I'm afraid. Is there any way via the developers console to alert/email if a pipeline fails? – Graham Polley

1 Answers

3
votes

There was a transient issue causing this failure which we've identified and should be resolved. We will be adding more checks/resilience to mitigate this issue from impacting dataflow jobs from progressing/shutting down in the future.

Thank you for the report, it was very helpful with our investigation.