Increase the number of Dask workers in Kubernetes deployment

Question

I have in Azure a Kubernetes cluster that deploys three Docker containers: a Flask server, a Dask Scheduler and a Dask Worker. This works fine, now I need to deploy N workers instead of one.

This is the yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: img-python
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cont-python
  template:
    metadata:
      labels:
        app: cont-python
    spec:
      containers:
        - name: cont-flask
          image: xxx.azurecr.io/myimage:latest
          command: ["gunicorn"]
          args: ["-t", "7200", "-b", "0.0.0.0:5000", "--reload", "app.front.app:app"]
          ports:
            - containerPort: 5000
          volumeMounts:
            - name: volcode
              mountPath: /code
        - name: cont-scheduler
          image: xxx.azurecr.io/myimage:latest
          command: ["dask-scheduler"]
          ports:
            - containerPort: 8786
            - containerPort: 8787
        - name: cont-worker
          image: xxx.azurecr.io/myimage:latest
          workingDir: /code
          command: ["dask-worker"]
          args: ["scheduler:8786"]
          volumeMounts:
            - name: volcode
              mountPath: /code
      volumes:
        - name: volcode
          azureFile:
            shareName: py-code
            readOnly: false

I looked at the kubectl scale command however it seems to scale the entire cluster, not only specifically the worker's container.

How to specify the number of workers?

danielorn danielorn · Accepted Answer · 2021-08-27T19:27:54

The yaml file specifies a deployment, which can be scaled horizontally to any number of pods (either by setting the spec.replicas in the yaml file or on the fly using kubectl scale)

It is not possible to scale individual containers inside a pod since pods are the smallest deployable units of computing that can be managed by Kubernetes

From the docs:

Each Pod is meant to run a single instance of a given application. If you want to scale your application horizontally (to provide more overall resources by running more instances), you should use multiple Pods, one for each instance. In Kubernetes, this is typically referred to as replication. Replicated Pods are usually created and managed as a group by a workload resource and its controller.

Thus, to scale out the number of workers to 10 without scaling the server and scheduler, the worker needs to be a separate deployment (so that the worker is run it its own pod).

Increase the number of Dask workers in Kubernetes deployment

1 Answers