Cannot use GPU in VM on GCP; GPU_ALL_REGIONS exceeded

0
votes

Trying to boot up a VM (notebook) with a GPU (any GPU type) in Google Cloud, I get this error:

"tensorflow-1-15-20210102-230326: Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally. (if this notebook instance needs to use a resource outside of project recsys-296821, in the project that contains the resource, please grant the missed permission to the Cloud AI Platform Notebooks Service Account: [email protected])".

I already tried to increase my total gpu quotas, having already upgraded my account from the Free tier (like in GCP error: Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally and GPU Quota Error even when i have Quota for Nvidia P100 both for region Us-west1 and Europe-west4 from 0 to 1 etc), but the GPU_ALL_REGIONS is grayed out and I can't change it, while still being set to 1, so in theory I should be able to use 1 gpu, but I cannot.

I really don't know how to solve this issue and couldn't find an up to date answer. Thanks in advance.

1
google-cloud-platform
Upgrading your account from Free Tier does not mean your GPU quota has been increased. You must ask for a quota increase. cloud.google.com/compute/quotas – John Hanley
ok, thank you, but I can't seem to be able to ask for a quota increase for all regions. Could you please tell me how to do it precisely? Following that guide I was apparently not able to, as for 'all regions' the option seemed grayed out. Thank you!!! – Filippo Fedeli
You request quota increases for a specific region, not globally. If your account is new, only ask for one region and one GPU. – John Hanley
ok, thanks. Looking at quotas in the regions, I see in theory I already have values different from zero for the regions, but then I cannot use them. I still tried to move one region from 1 to 2 as an experiment, but my request was promptly rejected. What have I done wrong? – Filippo Fedeli
GPUs are expensive. Is your account new? If yes, contact Google Cloud Billing to increase your credit (reduce restrictions). If you have GPU quota of 1 in a specific region then launch your services in that region. If you have a problem, create a detailed question showing your work, the problem, errors, etc. For quota related problems you will need to contact Google Cloud Support. – John Hanley

1 Answers

0
votes

According to this GCP official doc, I understand that you might have enough quota to use the resources but it is possible that there might not be available resources in that region. For example, you might still have quota to create VM instance with GPU in us-central1, but there might not be available GPU in that region.

Best possible solutions are:

1- This is a temporary situation that can change anytime based on fluctuating demand, try your request again later.

2- Try to create resources in another zone in the region or in another region.

3- Try to change the shape of the VM you are requesting. For example, you need NVidia P100 at this time, if it works you can use another Nvidia T4

4- Use Compute Engine reservations to reserve the resources within a zone to ensure that the resources you need are available when you need them.