Kubernetes HPA wrong metrics?

Question

I've created a GKE test cluster on Google Cloud. It has 3 nodes with 2 vCPUs / 8 GB RAM. I've deployed two java apps on it

Here's the yaml file:

apiVersion: apps/v1            
kind: Deployment
metadata:                    
  name: myapi           
spec:
  selector:                                                                          
    matchLabels:
      app: myapi
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: myapi
    spec:
      containers:
      - image: eu.gcr.io/myproject/my-api:latest
        name: myapi
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: myapi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myfrontend
spec:
  selector:
    matchLabels:
      app: myfrontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: myfrontend
    spec:
      containers:
      - image: eu.gcr.io/myproject/my-frontend:latest
        name: myfrontend
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: myfrontend
---

Then I wanted to add a HPA with the following details:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: myfrontend
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myfrontend
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 50
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: myapi
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapi
  minReplicas: 2
  maxReplicas: 4
  targetCPUUtilizationPercentage: 80
---

If I check kubectl top pods it shows some really weird metrics:

NAME                         CPU(cores)   MEMORY(bytes)   
myapi-6fcdb94fd9-m5sh7      194m         1074Mi          
myapi-6fcdb94fd9-sptbb      193m         1066Mi          
myapi-6fcdb94fd9-x6kmf      200m         1108Mi          
myapi-6fcdb94fd9-zzwmq      203m         1074Mi          
myfrontend-788d48f456-7hxvd   0m           111Mi           
myfrontend-788d48f456-hlfrn   0m           113Mi

HPA info:

NAME        REFERENCE              TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
myapi      Deployment/myapi      196%/80%   2         4         4          32m
myfrontend   Deployment/myfrontend   0%/50%     2         5         2          32m

But If I check uptime on one of the nodes it shows a less lower value:

[myapi@myapi-6fcdb94fd9-sptbb /opt/]$ uptime
 09:49:58 up 47 min,  0 users,  load average: 0.48, 0.64, 1.23

Any idea why it shows a completely different thing. Why hpa shows 200% of current CPU utilization? And because of this it uses the maximum replicas in idle, too. Any idea?

See note about outdated version of HorizontalPodAutoscaler in edited answer. — weibeld

weibeld weibeld · Accepted Answer · 2020-03-23T10:29:44

The targetCPUUtilizationPercentage of the HPA is a percentage of the CPU requests of the containers of the target Pods. If you don't specify any CPU requests in your Pod specifications, the HPA can't do its calculations.

In your case it seems that the HPA assumes 100m as the CPU requests (or perhaps you have a LimitRange that sets the default CPU request to 100m). The current usage of your Pods is about 200m and that's why the HPA displays a utilisation of about 200%.

To set up the HPA correctly, you need to specify CPU requests for your Pods. Something like:

      containers:
      - image: eu.gcr.io/myproject/my-api:latest
        name: myapi
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: myapi
        resources:
          requests:
            cpu: 500m

Or whatever value your Pods require. If you set the targetCPUUtilizationPercentage to 80, the HPA will trigger an upscale operation at 400m usage, because 80% of 500m is 400m.

Besides that, you use an outdated version of HorizontalPodAutoscaler:

Your version: v1
Newest version: v2beta2

With the v2beta2 version, the specification looks a bit different. Something like:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapi
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapi
  minReplicas: 2
  maxReplicas: 4
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

See examples.

However, the CPU utilisation mechanism described above still applies.

Kubernetes HPA wrong metrics?

1 Answers