Spring Cloud Dataflow - how to pass credentials to task

Question

I use spring cloud dataflow deployed to pivotal cloud foundry, to run spring batch jobs as spring cloud tasks, and the jobs require aws credentials to access an s3 bucket.

I've tried passing the aws credentials as task properties, but the credentials are showing up in the task's log files as arguments or properties. (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-global-properties)

For now, I am manually setting the credentials as env variables in pcf after each deployment, but I'm trying to automate this. The tasks aren't deployed until the tasks are actually launched, so on a deployment I have to launch the task, then wait for it to fail due to missing credentials, then set the env variable credentials with the cf cli. How so I provide these credentials, without them showing in the pcf app's logs?

I've also explored using vault and spring cloud config, but again, I would need to pass credentials to the task to access spring cloud config.

Thanks!

Answer 1

Here's a Task/Batch-Job example.

This App uses spring-cloud-starter-aws. And this starter already provides the Boot autoconfiguration and the ability to override AWS creds as Boot properties.

You'd override the properties while launching from SCDF like:

task launch --name S3LoaderJob --arguments "--cloud.aws.credentials.accessKey= --cloud.aws.credentials.secretKey= --cloud.aws.region.static= --cloud.aws.region.auto=false"

You can decide to control the log-level of the Task, so it doesn't log them in plain text.

Answer 2

Secure credentials for tasks should be configured either via environment variables in your task definition or by using something like Spring Cloud Config Server to provide them (and store them encrypted at rest). Spring Cloud Task stores all command line arguments in the database in clear text which is why they should not be passed that way.

Answer 3

After considering the approaches included in the provided answers, I continued testing and researching and concluded that the best approach is to use a Cloud Foundry "User Provided Service" to supply AWS credentials to the task.

https://docs.cloudfoundry.org/devguide/services/user-provided.html

Spring Boot auto-processes the VCAP_SERVICES environment variable included in each app's container.

http://engineering.pivotal.io/post/spring-boot-injecting-credentials/

I then used properties placeholders in the application-cloud.properties to map the processed properties into spring-cloud-aws properties:

cloud.aws.credentials.accessKey=${vcap.services.aws-s3.credentials.aws_access_key_id}
cloud.aws.credentials.secretKey=${vcap.services.aws-s3.credentials.aws_secret_access_key}