DataFlow pipeline: Project has insufficient quota

1
votes

I am following this to create a dataflow pipeline https://codelabs.developers.google.com/codelabs/modern-data-pipeline-firestore-bigquery-dataflow-templates/index.html#9 but my data flow failed due to my quota.

i got this error message: "Workflow failed. Causes: Project XXXX has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/7 instances, 4/4 CPUs, 1230/818 disk GB, 0/250 SSD disk GB, 1/99 instance groups, 1/49 managed instance groups, 1/99 instance templates, 1/3 in-use IP addresses. Please see https://cloud.google.com/compute/docs/resource-quotas about requesting more quota."

I was denied for quota increase by gcs, so I guess I'll have to optimize my pipeline by specifying the parameters. How should I do that?

1
google-cloud-firestoregoogle-cloud-dataflow
For exactly what quota did you requested the increase? – Louis C
You could try and contacting GCP Support, they could give you alternatives and more info on why the Quota increase was denied. – Louis C

1 Answers

0
votes

You should limit the maximum number of workers, the job will last a little bit more to finish but you are not going to hit the quotas. You can take a look at this document 1 2 on how to do so, the value you must change to limit the autoscale is "maxNumWorkers".