What is requesting all the cpu in my k8s cluster?

Question

Is there a way to see what pods/containers are requesting (not using) all the cpu on a node (or across all nodes in the cluster)?

We keep getting "insufficient cpu" errors when deploying to our cluster, even though I've turned down the cpu requests to sensible amounts for all our apps, and there is barely any cpu actually being used. By my calcs, my deployed containers are requesting around 400m. That's < 7% of the total available cpu (3 nodes x 2 cores = 6000m, right?). Yet the dashboard tells me each node's CPU requests are close to 2 cores (80-90%)

To me, it seems like other things are requesting too much cpu (ingress? logstash? vault? vault sidecars? metrics server? the k8s management layer?). Is there a way (eg. via kubectl or the dashboard) to see where it's all being requested?

Answer 1

For a single node, it's in the extended output of kubectl describe node node-name.

Remember that a pod is scheduled on a single node; it can't be split across nodes. If you have a pod that requires 100% CPU, and 3x 2-core nodes that are at 120% CPU utilization, then no single node has enough capacity to schedule the pod, even though the cluster as a whole has 3x 80% = 240% CPU available.

My general experience has been that it's easier to schedule pods on fewer larger nodes than more smaller nodes. Partly this is overhead from processes like the kubelet and any DaemonSet pods; partly this is the fragmentation issue I described in the previous paragraph. You can also artificially lower the resource requests for a pod to make it get scheduled, but it's prone to either getting throttled (for CPU usage) or evicted (for memory usage) if it's busier than you say.