Kubernetes autoscaling policies not working

1
votes

I've added some scaling policies to my HorizontalPodAutoscaler but they are not being applied. The scaleUp and scaleDown behaviours are being ignored. I need a way to stop pods scaling up and down every few minutes in response to small CPU spikes. Ideally the HPA would scale up quickly in response to more traffic but scale down slowly after about 30 minutes of reduced traffic.

I'm running this on an AWS EKS cluster and I have setup the policies according to https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior.

Could this be a limitation of EKS or my K8s version which is 1.14. I have run kubectl api-versions and my cluster does support autoscaling/v2beta2.

My Helm spec is:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ template "app.fullname" . }}
  labels:
    app: {{ template "app.name" . }}
    chart: {{ template "app.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: "{{ template "app.fullname" . }}-server"
  minReplicas: {{ .Values.hpa.minReplicas }}
  maxReplicas: {{ .Values.hpa.maxReplicas }}
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: AverageValue
        averageValue: 200m
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 300
    scaleDown:
      stabilizationWindowSeconds: 1200
      policies:
      - type: Pods
        value: 1
        periodSeconds: 300
3
kuberneteskubernetes-helmautoscalingpolicieskubernetes-hpa
Could you edit your question with the following info: 1. What is the expected behavior that you want to achieve? 2. What is the current one now? – Wytrzymały Wiktor
can you please make sure you have metric server installed ? in case you don't have it installed let me know I'll post an answer for how to do that right aws.amazon.com/premiumsupport/knowledge-center/… – Mickey Hovel
I do have metric server installed – Mikhail Janowski
Does the autoscaling/v2beta2 API differ on different Kubernetes versions? In the documentation it says "Starting from v1.18 the v2beta2 API allows scaling behavior to be configured through the HPA behavior field." – Mikhail Janowski
@MikhailJanowski As stated: Starting from v1.18. Your current version of K8s is 1.14. This will not work even if you have autoscaling/v2beta2 enabled. The easiest way out of it would be to upgrade to 1.18. – Wytrzymały Wiktor

3 Answers

1
votes

It works for me,

Client Version: v1.20.2 Server Version: v1.18.9-eks-d1db3c

kubectl api-versions and my cluster also supports autoscaling/v2beta2

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ template "ks.fullname" . }}-keycloak
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ template "ks.fullname" . }}-keycloak
  minReplicas: {{ .Values.keycloak.hpa.minpods }}
  maxReplicas: {{ .Values.keycloak.hpa.maxpods }}
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: {{ .Values.keycloak.hpa.memory.averageUtilization }}
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: {{ .Values.keycloak.hpa.cpu.averageUtilization }}
  behavior:
    scaleDown:
      stabilizationWindowSeconds: {{ .Values.keycloak.hpa.stabilizationWindowSeconds }}
      policies:
        - type: Pods
          value: 1
          periodSeconds: {{ .Values.keycloak.hpa.periodSeconds }}
{{- end }}
0
votes

As already discussed in the comments, even with autoscaling/v2beta2 enabled this function will not work on version 1.14.

Starting from v1.18 the v2beta2 API allows scaling behavior to be configured through the HPA behavior field.

The easiest way out of it would be to upgrade to 1.18.

0
votes

i wanted to implement HPA based on targetAverageValue what worked for me

while GKE version is around 1.12 to 1.14 you wont be able to apply manifest of autoscaling/v2beta2 however you can apply same thing something like

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: core-deployment
  namespace: default
spec:
  maxReplicas: 9
  minReplicas: 5
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: core-deployment
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageValue: 500m