Spring Cloud Task arguments are not propagated to Spring Batch job parameters

Question

I am migrating from plain spring batch to spring cloud task and spring batch executed through spring cloud data flow. I am experiencing the below problem in both my job and the example job that I have deployed for file injest:

https://docs.spring.io/spring-cloud-dataflow-samples/docs/current/reference/htmlsingle/#_batch_file_ingest

I am using the spring-cloud-dataflow-server-local-1.4.0.RELEASE with a postgres backend. I have also adapted the example to work with the postgres database.

The problem occurs when I attempt to run the task a second time with a different argument. When I run it the first time:

task launch fileInjectTask --arguments "filePath=classpath:data.csv --spring.cloud.task.closecontext_enable=false"

The following is the output in the logs:

2018-03-22 10:10:51.446 INFO 10431 --- [ main] o.s.b.a.b.JobLauncherCommandLineRunner : Running default command line with: [filePath=classpath:data.csv, --spring.cloud.task.closecontext_enable=false, --spring.cloud.task.executionid=13]

2018-03-22 10:10:51.497 INFO 10431 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ingestJob]] launched with the following parameters: [{filePath=classpath:data.csv, -spring.cloud.task.executionid=13, -spring.cloud.task.closecontext_enable=false, run.id=1}]

So that's fine, the parameter gets passed to the batch job. Now, when I attempt to run the same task with a different argument to the task:

task launch fileInjectTask --arguments "filePath=/home/hmcmanus/spring-cloud-dataflow-samples/batch/file-ingest/src/main/resources/data.csv --spring.cloud.task.closecontext_enable=false"

I get the following in the logs:

2018-03-22 10:12:18.249 INFO 10554 --- [ main] o.s.b.a.b.JobLauncherCommandLineRunner : Running default command line with: [filePath=/home/hmcmanus/spring-cloud-dataflow-samples/batch/file-ingest/src/main/resources/data.csv, --spring.cloud.task.closecontext_enable=false, --spring.cloud.task.executionid=14]

2018-03-22 10:12:18.322 INFO 10554 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ingestJob]] launched with the following parameters: [{filePath=classpath:data.csv, -spring.cloud.task.executionid=13, run.id=1, -spring.cloud.task.closecontext_enable=false}]

As you can see the second example the arguments to the task are fine, however the arguments to the batch job are the old arguments of the task execution.

Is there something that I'm missing here? Why doesn't the spring batch job get the parameters of the new task execution?

Update

To rule out differences in database and any modifications that I needed to make in order for it to run I have also verified the same functionality with the embedded H2 database for both the data flow server and the task.

And your job runs successfully with correct execution ids when cloud flow server is not being used? — Sabir Khan
Did your previous execution of that job with the same parameters complete successfully? — Michael Minella
@SabirKhan - Yup, plain old maven with passing arguments in the command line: mvn clean spring-boot:run -Dspring.cloud.task.closecontext_enable=false -Dspring-boot.run.arguments="filePath=classpath:data.csv" ``` Works fine. — hughmcmanus
@MichaelMinella - I wasn't sure about that so I ran it again, yes the previous job ran successfully, then run again with a different argument for the file path and the task picks it up but the spring batch job has the old parameter. — hughmcmanus

Matías Bustos Matías Bustos · Accepted Answer · 2018-03-22T02:08:03

I ran into the same situation with that example. So far, I have realized that jobs are being executed using the first execution's id of the task as a parameter, in your case task.executionid=13. This value is not changed along the job executions. According to Spring Cloud Task Reference:

Spring Boot provides facilities for the execution of batch jobs easily within an über-jar. Spring Boot’s support of this functionality allows for a developer to execute multiple batch jobs within that execution. Spring Cloud Task provides the ability to associate the execution of a job (a job execution) with a task’s execution so that one can be traced back to the other.

This functionality is accomplished by using the TaskBatchExecutionListener. By default, this listener is auto configured in any context that has both a Spring Batch Job configured (via having a bean of type Job defined in the context) and the spring-cloud-task-batch jar is available within the classpath. The listener will be injected into all jobs.

https://docs.spring.io/spring-cloud-task/current-SNAPSHOT/reference/htmlsingle/#batch-association

However, in the example the requirements are covered as stated.

Spring Cloud Task arguments are not propagated to Spring Batch job parameters

1 Answers