The Scenario: I have deployed a service using helm chart, I can see my service, hpa, deployment, pods etc. In my hpa setting: the min pod count is set to 1. I can see my Pod is running and able to handle service request.
After a while --- I have executed -- "kubectl scale deploy --replicas=0" Once I run the above above command I can see my pod got deleted (although the hpa min pod setting was set to 1), I was expecting after a while hpa will scale up to the min pod count i.e. 1. However I don't see that happened, I have waited more than an hour and no new pod created by hpa. I have also tried sending a request to my Kubernetes service and I was thinking now hpa will scale up the pod, since there is no pod to serve the request, however the hps doesn't seem to do that, and I got a response that my Service is not available.
Here is what I can see in kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE** test Deployment/xxxx /1000% 1 4 0 1h
Interestingly I found that hpa scale down quickly: when I execute "kubectl scale deploy --replicas=2" (please note that the in hpa count is 1), I can see 2 pods gets created quickly however within 5 mins, 1 pod gets removed by hpa.
Is this is expected behavior of Kubernetes (particularly hpa) ? as in, if we delete all pods by executing --"kubectl scale deploy --replicas=0", a) the hpa won't block to reduce the replica count less than pod count configured (in hpa config) and b) the hpa won't scale up (based on the hpa spinning cycle) to the min number of pods as configured. and essentially c) until we redeploy or execute another round of "kubectl scale deploy" to update the replica count there will be no pods for this service.
Is this expected behavior or a (possible) bug in the Kubernetes codebase ? I am using Kubernetes 1.8 version.