Since turning autoscaling on some of our Cloud Dataflow jobs have started failing. I am getting the following error on my failed job:
This is the job id for this error: 2016-01-26_12_02_19-12737268934091661877
I also have this job that has taken more then 5 hours when it usually takes 1 hour:
2016-01-27_12_02_52-11776322710903149175
EDIT: I turned autoscaling off and the job that ran this morning still ran really slow. Almost 5 hours.
The job id is: 2016-01-28_12_03_39-5692230370814846330
When writing to BigQuery, Dataflow first writes all your data to GCS and then runs a BigQuery import job (See this thread for some more details).
The error message from the first job indicates that there were problems with the BigQuery import job. The latency in the other jobs seems to be related to BigQuery jobs as well. In these cases, you should be able to find a message that includes the BigQuery job ID as well as a command to check its status.
We believe these issues should be resolved. You should be able to re-enable autoscaling without issue. Please try running the jobs again and let us know if you continue having problems.
