Auto scale kubernetes pods based on downstream api results

Question

I have seen HPA can be scaled based on CPU usage. That is super cool.

However, the scenario I have is: the stateful app (container in pod) is one to one mapping based on the downstream API results. For example, the downstream api results return maximum and expected capacity like {response: 10}. I would like to see replicaSet or statefulSet or other kubernetes controller can obtain this value and auto scale the pods to 10. Unfortunately, the pod replicas is hardcoded in the yaml file.

If I am doing it manually, I think I can do it via running start a scheduler. The job of the scheduler is to watch the api and run the kubectl scale command based on the downstream api results. This can be error prone and there is another system I need to maintain. I guess this logic should belong to a kubernetes controller ?

May I ask has someone done this stuff before and what is the way to configure it ?

Thanks in advance.

Answer 1

Unfortunately, it is not possible to use an HPA in that mode, but your conception about how to scale is right.

HPA is designed to analyze metrics and decide how many pods need to be spawned based on those metrics. It is using scaling rules and can only spawn pods one by one based on the result of its decision.

Moreover, it using standard Kubernetes API for scale pods.

Because a logic of HPA is already in your application, you can use the same API to scale your pods. Btw, kubectl scale is using the same way to interact with a cluster.

So, you can use i.e. Cronjob, with a small application which will call API of your application every 5 minutes and call kubectl scale with proper name of deployment to scale your app.

But, please keep in mind, you need to somehow control the frequency of up- and downscaling of pods, it will make your application more stable. That’s why I think that scaling not more often than once per 5 minutes is OK, but trying to do it every minute generally is not the best idea.

And of course, you can create a daemon and run it using Deployment, but I think Cronjob solution is more easy and faster to implement.