Google Kubernetes is not auto scaling to 0

Question

I am testing the Google Kubernetes autoscaling.

I have created a cluster with 1 master node.

Then I have used

gcloud container node-pools create node-pool-test \
--machine-type g1-small --cluster test-master \
--num-nodes 1 --min-nodes 0 --max-nodes 3 \
--enable-autoscaling --zone us-central1-a

to create a node pool with autoscaling and minimum node to 0.

Now, the problem is that it's been 30 minutes since the node pool was created (and I haven't run any pods) but the node pool is not scaling down to 0. It was supposed to scale down in 10 minutes.

Some system pods are running on this node pool but the master node is also running them.

What am I missing?

Answer 1

Have a look at the documentation:

If you specify a minimum of zero nodes, an idle node pool can scale down completely. However, at least one node must always be available in the cluster to run system Pods.

and also check the limitations here and here:

Occasionally, cluster autoscaler cannot scale down completely and an extra node exists after scaling down. This can occur when required system Pods are scheduled onto different nodes, because there is no trigger for any of those Pods to be moved to a different node

and possible workaround.

More information you can find at Autoscaler FAQ.

Also, as a solution, you could create one node pool with a small machine for system pods, and an additional node pool with a big machine where you would run your workload. This way the second node pool can scale down to 0 and you still have space to run the system pods. Here you can find an example.