Doesn't Kubernetes honor HPA configuration when we execute “kubectl scale deploy”?

2
votes

The Scenario: I have deployed a service using helm chart, I can see my service, hpa, deployment, pods etc. In my hpa setting: the min pod count is set to 1. I can see my Pod is running and able to handle service request.

After a while --- I have executed -- "kubectl scale deploy --replicas=0" Once I run the above above command I can see my pod got deleted (although the hpa min pod setting was set to 1), I was expecting after a while hpa will scale up to the min pod count i.e. 1. However I don't see that happened, I have waited more than an hour and no new pod created by hpa. I have also tried sending a request to my Kubernetes service and I was thinking now hpa will scale up the pod, since there is no pod to serve the request, however the hps doesn't seem to do that, and I got a response that my Service is not available.

Here is what I can see in kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE** test Deployment/xxxx /1000% 1 4 0 1h

Interestingly I found that hpa scale down quickly: when I execute "kubectl scale deploy --replicas=2" (please note that the in hpa count is 1), I can see 2 pods gets created quickly however within 5 mins, 1 pod gets removed by hpa.

Is this is expected behavior of Kubernetes (particularly hpa) ? as in, if we delete all pods by executing --"kubectl scale deploy --replicas=0", a) the hpa won't block to reduce the replica count less than pod count configured (in hpa config) and b) the hpa won't scale up (based on the hpa spinning cycle) to the min number of pods as configured. and essentially c) until we redeploy or execute another round of "kubectl scale deploy" to update the replica count there will be no pods for this service.

Is this expected behavior or a (possible) bug in the Kubernetes codebase ? I am using Kubernetes 1.8 version.

1
kuberneteskubectl

1 Answers

6
votes

That was great observation. I was going through documentation of HPA and come across mathematical formula used by HPA to scale pods .and it looks like 

TargetNumOfPods = ceil(sum(CurrentPodsCPUUtilization) / Target)

In your case, current pod utilization is zero as your pods count is zero . So mathematically this equation result into zero. So this this is a reason HPA is not working if pod count is zero.

a: HPA should not block manual scaling of pods as it get trigger only from resources (cpu, memory etc). Once you do scaling using "kubectl scale" or by any other means then HPA will come into picture depending on min, max replica and avg utilization value.

b: HPA scales up to min number of replicas if current count is non zero. I tried it and its working perfectly fine.

c: Yes unless you bring replica count to non-zero value, HPA will not work. So you have to scale up to some non zero value.

Hope this answers your doubts about HPA.