Conditionally launch Spring Cloud Task on a specific node of Kubernetes cluster

Question

I am building a data pipeline for batch processing. And I find that Spring Cloud Data Flow is a quite attractive framework to use. Without much knowledge in SCDF and Kubernetes, I am not sure whether it is possible to conditionally launch a Spring Cloud Task on a specific machine.

Suppose I have two physical servers that are for running the batch process (Server A and Server B). By default, I would like my Spring cloud task to be launched on Server A. If the Server A is shut down, the task should be deployed on server B. Can Kubernetes / SCDF handle this kind of mechanism? I am wondering whether the nodeselector is the thing that I should look into.

Answer 1

Yes, you can pass deployment.nodeSelector as a deployment property when launching the task.

The deployment.nodeSelector is a Kubernetes deployment property and hence, you need to pass something like this:

task launch mytask --properties "deployer.<taskAppName>.kubernetes.deployment.nodeSelector=foo1:bar1,foo2:bar2"

You can check the list of supported Kubernetes deployer properties here