GKE: How to handle deployments with CPU intensive initialization?

5
votes

I have a GKE cluster (n1-standard-1, master version 1.13.6-gke.13) with 3 nodes on which I have 7 deployments, each running a Spring Boot application. A default Horizontal Pod Autoscaler was created for each deployment, with target CPU 80% and min 1 / max 5 replicas.

During normal operation, there is typically 1 pod per deployment and CPU usage at 1-5%. But when the application starts, e.g after performing a rolling update, the CPU usage spikes and the HPA scales up to max number of replicas reporting CPU usage at 500% or more.

When multiple deployments are started at the same time, e.g after a cluster upgrade, it often causes various pods to be unschedulable because it's out of CPU, and some pods are at "Preemting" state.

I have changed the HPAs to max 2 replicas since currently that's enough. But I will be adding more deployments in the future and it would be nice to know how to handle this correctly. I'm quite new to Kubernetes and GCP so I'm not sure how to approach this.

Here is the CPU chart for one of the containers after a cluster upgrade earlier today:

CPU usage

Everything runs in the default namespace and I haven't touched the default LimitRange with 100m default CPU request. Should I modify this and set limits? Given that the initialization is resource demanding, what would the proper limits be? Or do I need to upgrade the machine type with more CPU?

1
spring-bootkubernetesgoogle-kubernetes-engineautoscaling
how long does the initialization burst take? Have you considered changing the deployment rollout parameter to be maxSurge: 1 and minReadySeconds: 60 so that it will only surge one pod per 60 seconds? This way while you have slower rollouts, you can do them without needing a handful of spare nodes. – eamon1234
Thanks for the suggestion.The Spring Boot applications typically take around 30 seconds to initialize. It seems maxSurge already defaults to 1, but I tried changing minSecondsReady from 0 to 60 and unfortunately it didn't seem to help as the CPU still burst and the HPA scaled up. – André
with two replicas and say 50% CPU threshold, the HPA is always going to go from one to two if one replica is at 100%. But with 10 replicas and maxSurge of one, with 50% threshold, it shouldn't. – eamon1234
Can you configure the readiness probe to only start once the CPU usage has dropped? Either by setting a probe that can only start succeeding after the initialization or by setting the "initialDelaySeconds" so that the probes only start one the CPU usage has come down. The HPA should not take unready pods into consideration – Patrick W
@PatrickW It seems readiness probes is exactly what I need. Will look more into that, thanks! – André

1 Answers

3
votes

HPA only takes into account ready pods. Since your pods only experience a spike in CPU usage during the early stages, your best bet is to configure a readiness probe that only shows as ready once the CPU usage comes down or has a initialDelaySeconds set longer than the startup period to ensure the spike in CPU usage is not taken into account for the HPA.