Why kubernetes HPA is not scalling down pods?

Question

I am using prometheus and adapter to scale HPA (custom metrics memory_usage_bytes). I don't know why m is appended with targetValue and also HPA does not scaled down pods when they don't use memory.

Am i missing anything?

HPA code

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: pros
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: pros
  maxReplicas: 3
  metrics:
  - type: Pods
    pods:
      metricName: memory_usage_bytes
      targetAverageValue: 33000000

kubectl get hpa

NAME   REFERENCE         TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
pros   Deployment/pros   26781013333m/33M      1         3         3          19m

custom.metrics.k8.io

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/memory_usage_bytes"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "pros-6c9b9c5c59-57vmx",
        "apiVersion": "/v1"
      },
      "metricName": "memory_usage_bytes",
      "timestamp": "2019-07-13T12:03:10Z",
      "value": "34947072",
      "selector": null
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "pros-6c9b9c5c59-957zv",
        "apiVersion": "/v1"
      },
      "metricName": "memory_usage_bytes",
      "timestamp": "2019-07-13T12:03:10Z",
      "value": "19591168",
      "selector": null
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "pros-6c9b9c5c59-nczqq",
        "apiVersion": "/v1"
      },
      "metricName": "memory_usage_bytes",
      "timestamp": "2019-07-13T12:03:10Z",
      "value": "19615744",
      "selector": null
    }
  ]
}

mario mario · Accepted Answer · 2019-07-23T11:29:59

There are at least two good reasons explaining why it may not work:

As you can see in documentation:

The current stable version, which only includes support for CPU autoscaling, can be found in the autoscaling/v1 API version. The beta version, which includes support for scaling on memory and custom metrics, can be found in autoscaling/v2beta2.

and you are using:

apiVersion: autoscaling/v2beta1 in your HorizontalPodAutoscaler definition.
If you sum up total memory used by all 3 currently running pods from your custom.metrics.k8.io example, the workload still won't fit on just 2 Pods when memory limit is set to 33000000. Notice that first Pod have already reached its limit of 33M and memory consumption by other 2 Pods (19591168 + 19615744) is still too high to fit it on a single pod with the 3300000 limit.

Why kubernetes HPA is not scalling down pods?

1 Answers