Distributed CPU-intensive processes in Kubernetes/Docker

Question

I have an application that uses a lot of CPU. I thought in Kubernetes as a way to distribute the workload among small pieces of work, then I created several CPU-limited pods. It turns out that Docker has a constraint in which it distributes the total amount of CPU between all the containers running CPU-intensive processes (https://docs.docker.com/engine/reference/run/#cpu-share-constraint). For that reason, each pod can not use the whole amount of CPU that it should have since Docker share out resources by itself.

Example:

Environment: Kubernetes Platform with 80 CPU cores available across the cluster

Test1:

• Context: 1 single pod limited to 5 CPU cores
• Processes: 1 single process running in the single pod
• Duration: the single process lasts 0:02:05

Test2:

• Context: 12 pods limited to 5 CPU cores each
• Processes: 12 processes running each on every pod
• Duration: it takes an average of 0:03:55 to process each one

This means that the CPU usage is affected (and then the processing time increases) when there are several containers requesting CPU resources.

I guess that Docker is not intended to be used as I need to.

I understand that in this scenario it would be better to use VM's instead of Docker containers, but Is there a way to make it work (maybe changing Docker or Kubernetes configuration)?

Any helpful comment would be appreciated.

Answer 1

Its hard to give an exact answer since we don't know more about your cluster set-up and what you are actually processing. However, I believe the difference is because the CPU shares constraint is not doing what you think it is.

From the docker documentation that you linked:

The proportion will only apply when CPU-intensive processes are running. When tasks in one container are idle, other containers can use the left-over CPU time.

This means that in you first test, even though you restricted the CPU-shares to 5, IF there is nothing else running on that host, it will continue to grow in CPU usage until 100% of CPU time on that host is used up.

In the 12 processor case, each of those containers is in fact competing with the others on the host for CPU resources, and therefore, only receiving a portion of the whole host system CPUs. They will each receive an equal share of total CPU time, but that share would still be less than receiving all of the CPU time.

Rather than using CPU-shares to limit the resources, you could use the --cpu's flag. This will give you the cpu usage restrictions that I believe you were actually trying to achieve.

Specify how much of the available CPU resources a container can use. For instance, if the host machine has two CPUs and you set --cpus="1.5", the container will be guaranteed to be able to access at most one and a half of the CPUs. This is the equivalent of setting --cpu-period="100000" and --cpu-quota="150000". Available in Docker 1.13 and higher.

Docker Resource Constraints Documentation

Answer 2

This might be too late, but I want to add an addition to user2566987's answer, in Kubernetes, there is something called cpu manager policy which by default will behave the like what the above answer said, but there is another policy which is static mode, which will reserve the CPU that you want by removing them from shared CPU pool, this will guarantee to the pod to get an exclusive CPU time without them being shared.

This requires the pod request and limit to be set equal ( Guaranteed Quality of Service class) and make they are integer number >= 1.