Dataflow not scaling

Question

I had a dataflow pipeline running on GCP with the max workers set to 20. This pipeline reads data from BigQuery and using that data to retrieve data from an Elasticsearch cluster. When running it, it scaled correctly to 20 workers but upon seeing that it was still taking to long to run, I cancelled the pipeline and restarted it with the max workers set to 30. For some reason however it's no longer scaling correctly. It's currently stuck at 9 workers and shows no sign of increasing. It's not even reaching the 20 workers it had before, even though it's the exact same pipeline

Anyone know what could cause this or how to debug this?

Answer 1

Without the job id it's hard to determine the root cause of this behaviour (please, don't share it here though). What you can try, however, is using more powerful machines for your workers (vertical scaling). If after trying again it doesn't work, I would recommend you to open a ticket with Google Cloud Platform Support.