kubernetes - resource limits exceeded

0
votes

I have a cluster running Kubernetes v1.6.7.

The VMs are :

Distributor ID: Debian Description: Debian GNU/Linux 8.7 (jessie) Release: 8.7 Codename: jessie

Kernel : 3.16.0-4-amd64

We have set cpu limits in deployments, however pods comsume cpu over this limit if they need to. Is there something am I missing ? Like a parameter in kubelet to enable this limit ? I haven't found anything about this problem.

For example, if I create the following deployment :

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: stress namespace: default labels: k8s-app: stress spec: replicas: 1 selector: matchLabels: k8s-app: stress template: metadata: labels: k8s-app: stress spec: containers: - name: stress image: progrium/stress imagePullPolicy: Always args: ["--cpu", "1"] resources: limits: cpu: "500m" memory: "1Gi" requests: cpu: "100m" memory: "512Mi"

On the node, this makes a CPU 100% used whereas it should be 50%.

Thanks for your help.

1
resourceskubernetescpu
Can you give more details about how you limiting CPU, how did you check that pods consume CPU over limit? - Anton Kostenko
Thanks for your reply, I edited my question with the details. - matth3o
Can you also tell how you checked CPU load? Top, or using some monitoring etc. - Anton Kostenko
With top and htop : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17096 root 20 0 7316 96 0 R 100,0 0,0 2:40.82 /usr/bin/stress --verbose --cpu 1 - matth3o
Kubernetes dashboard also tells me this pod has a cpu usage of 1 core. - matth3o

1 Answers

2
votes

From a comments to your question, where you can try to run docker container with CPU limit you getting: WARNING: Your kernel does not support CPU cfs period or the cgroup is not mounted. Period discarded. WARNING: Your kernel does not support CPU cfs quota or the cgroup is not mounted. Quota discarded.

That mean your system's kernel build without support of CFS. Here is an issue about it, right about 3.16.0-4-amd64 version.

You need to rebuild or update your kernel.

Here is how to update it from backports:

  • Add mirror with backports: $ sudo vi /etc/apt/sources.list.d/sources.list deb http://http.debian.net/debian jessie-backports main deb-src http://http.debian.net/debian jessie-backports main

  • Update apt: sudo apt-get update

  • Install a kernel from a backport repo:sudo apt-get install -t jessie-backports linux-image-amd64

  • Reboot after instalation.