GCP Dataflow job is stuck in "updating" status for 18 hours

1
votes

My Dataflow job on GCP (2019-05-07_06_51_21-705453987744722209) has been stuck in status "updating" for the past 18 hours. The job that is an update for this job has been in status "not started" for the same amount of time (2019-05-07_07_48_56-11394590249055546323). I can't cancel or drain either of the jobs.

I can't cancel through the console because there are no options to do so. I tried doing it from the command line (gcloud dataflow jobs cancel, etc...) but got the response:

Failed to cancel job [2019-05-07_06_51_21-705453987744722209]: (9d2be4fd48bdc9f): Workflow modification failed. Causes: (ed81ed4ba124b24d): Cannot perform operation 'cancel' on Job: 2019-05-07_06_51_21-705453987744722209. Job has action requested: RELOAD. Please ensure you have permission to access the job and the --region flag, europe-west1, matches the job's region.

I want to cancel both the original job and the updated job, but haven't been able to do so, any help or suggestions would be greatly appreciated.

1
google-cloud-platformgoogle-cloud-dataflow
Hi there, how did that turn out? I'm having the same issue. – Bruno Volpato

1 Answers

0
votes

This issue can happen when the Dataflow Manager and workers get out of sync or when jobs are stuck because of bugs in your Dataflow code.

I would suggest that you opened a ticket with Google Cloud Platform support so they can investigate the jobs and force the Dataflow Manager to give up the jobs.

Manuel