Worker pod resource limit in spring batch and spring cloud deployer kubernetes

Question

I am trying to run a Spring Batch application in kubernetes cluster. I am able to enforce resource limits to the main application pod by placing the following snippet in the deployment yaml:

resources:
  limits:
    cpu: 500m
    ephemeral-storage: 500Mi
    memory: 250Mi

These settings are getting applied and can be seen in the pod yaml (kubectl edit pod batch).

However, these limits are not propagated to the worker pods. I tried adding the following properties in configmap of batch to set the cpu and memory limits:

SPRING.CLOUD.DEPLOYER.KUBERNETES.CPU: 500m
SPRING.CLOUD.DEPLOYER.KUBERNETES.MEMORY: 250Mi

However, the worker pods are not getting these limits. I tried providing the following env variables too, but still the limits were not applied to the worker pod:

SPRING_CLOUD_DEPLOYER_KUBERNETES_CPU: 500m
SPRING_CLOUD_DEPLOYER_KUBERNETES_MEMORY: 250Mi

The versions involved are:

Spring Boot: 2.1.9.RELEASE
Spring Cloud: 2020.0.1
Spring Cloud Deployer: 2.5.0
Spring Cloud Task: 2.1.1.RELEASE
Kubernetes: 1.21

How can I set these limits?

EDIT: Adding code for DeployerPartitionerHandler:

public PartitionHandler partitionHandler(TaskLauncher taskLauncher, JobExplorer jobExplorer) {

    Resource resource = this.resourceLoader.getResource(resourceSpec);

    DeployerPartitionHandler partitionHandler = new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
            "worker");

    commandLineArgs.add("--spring.profiles.active=worker");
    commandLineArgs.add("--spring.cloud.task.initialize.enable=false");
    commandLineArgs.add("--spring.batch.initializer.enabled=false");
    commandLineArgs.add("--spring.cloud.task.closecontext_enabled=true");
    commandLineArgs.add("--logging.level.root=DEBUG");

    partitionHandler.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
    partitionHandler.setEnvironmentVariablesProvider(environmentVariablesProvider());
    partitionHandler.setApplicationName(appName + "worker");
    partitionHandler.setMaxWorkers(maxWorkers);

    return partitionHandler;
}

@Bean
public EnvironmentVariablesProvider environmentVariablesProvider() {
    return new SimpleEnvironmentVariablesProvider(this.environment);
}

Are you using the DeployerPartitionHandler to create workers? Please share your code to be able to help you efficiently. — Mahmoud Ben Hassine
I don't have any specific implementation of TaskLauncher for kubernetes. It is picking up the default TaskLauncher bean provided by spring cloud deployer kubernetes — Abhinav Sharma

Mahmoud Ben Hassine Mahmoud Ben Hassine · Accepted Answer · 2021-09-02T10:13:33

Since the DeployerPartitionHandler is created using the new operator in the partitionHandler method, it is not aware of the values from the properties file. The DeployerPartitionHandler provides a setter for deploymentProperties. You should use this parameter to specify deployment properties for worker tasks.

Worker pod resource limit in spring batch and spring cloud deployer kubernetes

1 Answers