Time-based scaling with Kubernetes CronJob: How to avoid deployments overriding minReplicas

3
votes

I have a HorizontalPodAutoscalar to scale my pods based on CPU. The minReplicas here is set to 5:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-web
  minReplicas: 5 
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

I've then added Cron jobs to scale up/down my horizontal pod autoscaler based on time of day:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: production
  name: cron-runner
rules:
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["patch", "get"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cron-runner
  namespace: production
subjects:
- kind: ServiceAccount
  name: sa-cron-runner
  namespace: production
roleRef:
  kind: Role
  name: cron-runner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-cron-runner
  namespace: production
---

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: django-scale-up-job
  namespace: production
spec:
  schedule: "56 11 * * 1-6"
  successfulJobsHistoryLimit: 0 # Remove after successful completion
  failedJobsHistoryLimit: 1 # Retain failed so that we see it
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cron-runner
          containers:
          - name: django-scale-up-job
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - kubectl patch hpa myapp-web --patch '{"spec":{"minReplicas":8}}'
          restartPolicy: OnFailure
----
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: django-scale-down-job
  namespace: production
spec:
  schedule: "30 20 * * 1-6"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 0 # Remove after successful completion
  failedJobsHistoryLimit: 1 # Retain failed so that we see it
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cron-runner
          containers:
          - name: django-scale-down-job
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - kubectl patch hpa myapp-web --patch '{"spec":{"minReplicas":5}}'
          restartPolicy: OnFailure

This works really well, except that now when I deploy it overwrites this minReplicas value with the minReplicas in the HorizontalPodAutoscaler spec (in my case, this is set to 5)

I'm deploying my HPA using kubectl apply -f ~/autoscale.yaml

Is there a way of handling this situation? Do I need to create some kind of shared logic so that my deployment scripts can work out what the minReplicas value should be? Or is there a simpler way of handling this?

1
kuberneteskubernetes-cronjobhpa
Hello, I'm having a difficulty to understand your question. Could you tell the whole scenario for the time based HPA scaling of yours? What exactly you mean by overwriting the minReplicas (as it should looking at your example)? You should create HPA that would be encompassing both situation (high load/low load) in a single HPA basing on the available metrics. – Dawid Kruk
Ah sorry - I've added the HPA config which should hopefully make it clearer – MDalt
If you would like to avoid overwriting the value of replicas in your Deployment (each new create/apply) you could try to update your Deployment with JSON merge patch. Is this what you are looking for? – Dawid Kruk
Ah...you mean each deployment you use a merge patch instead of apply, then this will only update values that have explicitly changed? Will give that a go and revert back if so, sounds smart. – MDalt
I've had a go with this and clarified the problem. Looks like it's definitely the HPA minReplicas value that's overwriting the one set by the CronJob (as opposed to the replicas in the Deployment). I tried using JSON merge to deploy the HPA (kubectl patch -f autoscale.yaml --type=merge -p "$(cat autoscale.yaml)") and it didn't work. Any other ideas? – MDalt

1 Answers

2
votes

I think you could also consider the following two options:

Use helm to manage the life-cycle of your application with lookup function:

The main idea behind this solution is to query the state of specific cluster resource (here HPA) before trying to create/recreate it with helm install/upgrade commands.

I mean to check the current minReplicas value each time before you upgrade your application stack.

Manage the HPA resource separately to application manifest files

Here you can handover this task to a dedicated HPA operator, which can coexist with your CronJobs that adjust minReplicas according specific schedule: