Google Cloud DataFlow Autoscaling not working

2
votes

I'm running a dataflow job that has 800K files to process. The job id is 2018-08-23_07_07_46-4958738268363865409.

It reports that it has successfully listed 800K files, but for some odd reason, the autoscaler only assigned 1 worker to it. Since it's processing rate is 2/sec, this is going to take a loooong time. I didn't touch the default scaler settings which to my knowledge means it can scale freely up to 100 workers. Why doesn't it scale?

Thanks,

Tomer

Update: Following Neri's suggestion, I started a new job (id 2018-08-29_13_47_04-1454220104656653184) and set autoscaling_algorithm=THROUGHPUT_BASED even though according to the documentation it should default to that anyway. Same behavior. processing speed is at 1 element per second and I have only one worker.

screenshot from web UI

What's the use of running in the cloud if you cannot scale?

1
google-cloud-dataflow
On Stackdriver Logging, what values do you see for autoscalingAlgorithm, numWorkers & maxNumWorkers? - Héctor Neri
In the worker startup: autoscalingAlgorithm=None. I found no entry in the logs for the DataFowStep with numWorkers / maxNumWorkers. - Tomer

1 Answers

0
votes

In order to autoscale your Dataflow Job, be sure that you use autoscalingAlgorithm = THROUGHPUT_BASED.

If you use "autoscalingAlgorithm":"NONE", then your Dataflow Job will get stuck even if it could autoscale. Otherwise, you will need to specify the number of workers you want on numWorkers.

Also, to scale to the amount of workers you want, be sure to specify (for numWorkers and maxNumWorkers) a number equal or lower to your quota, check your quota by using:

gcloud compute project-info describe