Why does gcloud ml-engine submit command give “requested cpu s exceed quota”?

1
votes

I am running a tensorflow object detection job on GCP with the folowing command:

gcloud ml-engine jobs submit training whoami_object_detection_date +%s --job-dir=gs://${YOUR_GCS_BUCKET}/train --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz --module-name object_detection.model_tpu_main --runtime-version 1.9 --scale-tier BASIC_TPU --region us-central1 -- --model_dir=gs://${YOUR_GCS_BUCKET}/train --tpu_zone us-central1 --pipeline_config_path=gs://${YOUR_GCS_BUCKET}/data/pinches_pipeline.config

Got the following error:

ERROR: (gcloud.ml-engine.jobs.submit.training) RESOURCE_EXHAUSTED: Quota failure for project seal-pinches. The requested 54.0 CPUs exceeds the allowed maximum of 20.0. To read more about Cloud ML Engine quota, see https://cloud.google.com/ml-engine/quotas. - '@type': type.googleapis.com/google.rpc.QuotaFailure violations: - description: The requested 54.0 CPUs exceeds the allowed maximum of 20.0.

My question is how the requested CPU getting set to 54? I am not setting this anywhere explicitly.

Thanks in advance.

1
pythontensorflowgoogle-cloud-platform

1 Answers

0
votes

This option in your code is setting the size and type of your ml instance:

--scale-tier BASIC_TPU

The BASIC_TPU costs $6.8474 per hour. I am not sure of the formula, but a Cloud TPU translates into N CPUs in equivalent billing. You also need to add the cost of the Cloud ML Engine machine type to your cost: standard is $0.2774 per hour.

Google's description:

Quota is defined in terms of Cloud TPU cores. A single Cloud TPU device comprises 4 TPU chips and 8 cores: 2 cores per TPU chip. A Cloud TPU v2 Pod (alpha) consists of 64 TPU devices containing 256 TPU chips (512 cores). The number of cores also specifies the quota for a particular Cloud TPU. For example, a quota of 8 enables the use of 8 cores. A quota of 16 enables use of up to 16 cores, and so forth.

Your CPU quota is 20. You will need to increase your quota or choose a different model such as BASIC or BASIC_GPU which does not use TPUs. Also double check that you have billing setup with a credit / debit card with sufficient credit available.