GCE (Google Compute Engine) GPU Quota >= 1 but starting VM fails

0
votes

I was working with a GPU in my GCE Project and decided to request a quota increase to 2. My VM instance-1 is in us-east1-c, so I navigated to IAM & Admin > Quotas and selected the following: Services - selected only Compute Engine API Metric - selected NVIDIA P100 GPUs & selected NVIDIA p100 Virtual Work Station GPUs Locations - selected only us-east1 After receiving notification that my increased quota to 2 GPUs had been approved, I went to start my VM instance and got the error code:

enter image description here

I double checked my quotas page and confirmed the increase to 2 GPUs. Support@google told me to request an incrase for GPUs Globaly, so I submitted a new request, selecting all, including the Global option:

enter image description here

Now when I attempt to start VM instance-1, I get:

enter image description here

So the error message says my quota is 0.0, however, looking at my Quotas under IAM & admin page, I have a quota of 2 in every location, including us-east1 (except for europe-west3, for some reason):

enter image description here

In an attempt to troubleshoot, I've tried to start new VM instances with GPUs in different locations in the same project, and also in a new project but got similar error messages each time. I've verified the locations I've chosen have GPUs using this link To add to the confusion, no one played with any settings in the new Project I started as a testing ground; the quotas were all at the default of 1, and I still got error messages. What am I doing wrong?

2
gpugoogle-compute-enginequota

2 Answers

1
votes

In addition to request a quota increase in the region you also need to request a Global quota increase.

Similar to virtual CPU quota, GPU quota refers to the total number of virtual GPUs in all VM instances in a region. Check the quotas page to ensure that you have enough GPUs available in your project, and to request a quota increase. In addition, new accounts and projects have a global GPU quota that applies to all regions.

When you request a GPU quota, you must request a quota for the GPU models that you want to create in each region, and an additional global quota for the total number of GPUs of all types in all zones.

This process is documented HERE

enter image description here You can use the filter Location = Global to find it

Now for the second error that the zone does not have enough resources, if you see 'ZONE_RESOURCE_POOL_EXHAUSTED' error meaning you are encountered with the temporary resource stock-out issue at that particular zone., you should try to deploy in a different zone

I advise you to remove your second image from this thread as your project ID is visible

0
votes

Your GPUs (all regions) count must equal the sum of GPUs in each region you have requested an increased quota. A search using ctrl & f did not work for me -- navigate to IAM & admin > click the Metric drop down > in the provided search bar, type "GPUs (all regions) "