Horizontal Pods Autoscaling (HPA) on GKE based on GPU utilization

2
votes

I am trying to achieve Horizontal Pods Autoscaling (HPA) in GCP based on GPU utilization. My hpa.yaml file looks like this:

kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
spec:
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
     metricName: kubernetes.io|container|accelerator|duty_cycle
      targetAverageValue: 10
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-container-worker

After I run kubectl create -f hpa.yaml , I see the following error in the GCP.

enter image description here

On the HPA, it says unable to read all metrics.

In stack driver monitoring I have created a dashboard with the metric mentioned above.

Has anyone come across this issue?

1
kubernetesgoogle-compute-enginegoogle-kubernetes-enginekubernetes-podhpa
try using compute.googleapis.com|container|accelerator|duty_cycle - Matt
that doesn't seem to be working either.. I get the same error - Suraj Acharya
Make sure you deployed Custom Metrics Stackdriver Adapter correctly and that its Running. I just created a gke cluster with hpa and gpus and all seems to be working for me. - Matt
Thank you.. It seem to be working now. I just need to figure out the targetAverageValue that works. What do you usually set it to? - Suraj Acharya
Now I get, this error when I run kubectl describe hpa Warning FailedComputeMetricsReplicas 43m horizontal-pod-autoscaler failed to get kubernetes.io|container|accelerator|duty_cycle external metric: unable to get external metric default/kubernete s.io|container|accelerator|duty_cycle/&LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[],}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request ( get kubernetes.io|container|accelerator|duty_cycle.external.metrics.k8s.io) - Suraj Acharya

1 Answers

0
votes

Summarizing our conversation in comments:

To make use of GCP metrics in kubernetes, Custom Metrics Stackdriver Adapter had to be deployed. This is why HPA was unable to read the metrics and therefore was throwing errors.