Google Cloud Dataflow 575 limit in central region

0
votes
Autoscaling: Unable to reach resize target in zone us-central1-b. QUOTA_EXCEEDED: Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 575.0 in region us-central1.

From https://cloud.google.com/dataflow/service/dataflow-service-desc my limit should be 1,000, but when running my dataflow job I get the warning above about a limit of only 575. Should I explicitly set a different region as specified in https://cloud.google.com/dataflow/docs/concepts/regional-endpoints or can I increase the limit to 1,000 in the central region?

1
google-cloud-dataflowapache-beam

1 Answers

2
votes

I guess you are using the default machine type, so each machine has one cpu but each of them has a standalone IP_ADDRESS. Even you can use up to 1000 instances, it seems your "IN_USE_ADDRESSES" quota in the region is set to 575 thus the error.

If you don't want to increase the number of "In Use Addresses", you can find other machine types to use more CPUs per instance, for example n1-standard-4. Otherwise, you can ask for more quota in "In Use Addresses".