Unable to Start gcloud instance with more GPUs despite increasing quota

Question

Relatively new to using gcloud suite and I'm having trouble starting my instance with more GPUs. Initially adding the first GPU was no issue in requesting a quota increase.

However, when I requested another increase to my GPU quota to improve the training time for my model, after getting approval and assigning another GPU to my instance, I am still receiving an error message that my instance cannot be started because it exceeded the allocated quota of 1 despite seeing in the admin console that the quota is 2.

Console:

Error:

Starting instance(s) my-fastai-instance...failed.
ERROR: (gcloud.compute.instances.start) Quota 'PREEMPTIBLE_NVIDIA_T4_GPUS' exceeded. Limit: 1.0 in region us-west1.

Is there something else I need to change in order to get the extra GPU to work with the instance?

Answer 1

Unfortunately, there's a distinction between "NVIDIA T4 GPUs" and "Preemptible NVIDIA T4 GPUs".

You've a quota of 2 for the "NVIDIA T4 GPUs" and -- I assume -- a quota of 1 for the "Preemptible NVIDIA T4 GPUs".

As you did before, you need to request these by region.

NB You should be able to proceed immediately by switching your instance from requesting "Preemptible NVIDIA T4 GPUs" to "NVIDIA T4 GPUs"