Google cloud dataflow job hangs

1
votes

Lately I had a few dataflow jobs hanging cases, when checked on the console the job was running but when click through that job it actually succeeded. After hours and days it stays that way and never exit, I had to kill the submitting jobs at backend so rest of my big job could procceed.

Some details:

project: smart-altar-543;

credential: the default service account of the project/VM Note, I suspect the hanging issue is because of the service account as I never had this issue when use personal account;

job_ID: 2015-06-21_09_22_15-13235319036709578079; 2015-06-20_23_10_28-1533493207141169764

1
google-cloud-dataflow
It seems that you have hit a bug in the Dataflow service. We're investigating and we'll update you with the results. - jkff
Note that this should be a very infrequent issue, it should not be related to using a service account, and it has already shut down the workers so it should not cause any billing implications. So I think you can submit other jobs in the meantime. - jkff
yes, did notice that workers shut down when jobs still hang, any cost implication to the dataflow service since the jobs are still running? - Echo
No. Pricing is currently based on Compute Engine pricing (see cloud.google.com/dataflow/pricing), and all Compute Engine resources are already released by this job. - jkff

1 Answers

1
votes

This issue is fixed now. There was an issue with metadata propagation in the service that has been remedied.