Spring Cloud Data Flow: versioned streams

1
votes

I'm implementing a stream pipe with Spring Cloud Data Flow.

My problem is that I configured MANUALLY the pipe (e.g. http | log_sink) in the server and it will be lost if I reset that server (think in an Amazon EC2 instance that can be hard reseted).

Which is the suggested way to keep versioning of pipes using SCDF?

Thanks.

1
spring-cloudspring-cloud-streamspring-cloud-dataflow
It'd help if you can clarify what do you mean by manual configurations and as well as the "hard reset" — not sure what actually is the problem — more details would help answer the question. – Sabby Anandan
Not sure I fully understand what you're saying. If your goal is to promote your stream/task workloads from dev to production or other environments, in a more CI/CD + automated manner, you can use SCDF's Java DSL library to programmatically register, define, deploy and do everything you can via Shell/UI. – Sabby Anandan
Docs: dataflow.spring.io/docs/feature-guides/streams/java-dsl; you can use also our DataFlowIT for reference, which we use to run IT tests on upstream commits. We have acceptance-tests running with similar infrastructure on many versions of K8s and Cloud Foundry daily. This automation maybe is what you're looking for. – Sabby Anandan
Oh, you totally can dump all of your register, stream or task commands in a text or a properties file. And, using the Shell command (i.e: dataflow:>script --file <text or property> file). – Sabby Anandan
This text file can be versioned in Git, as well. We have customers following this model in a production setting. I'd still recommend the DSL approach because you can build a standalone Boot app using the Java DSL, and version that app as a whole in Git. The app then becomes a reusable artifact (if parameterized and genericized sufficiently), and it will be CD-automation friendly. – Sabby Anandan

1 Answers

1
votes

I am summarizing the discussion from the comments.

To automate the promotion of Stream/Task workloads from lower to higher-level environments, the recommended approach would be the use of SCDF's Java DSL. With this, users can programmatically register, create, deploy, or launch stream/task in a repeatable manner and across many different platforms simultaneously (if there's a need for it). The Boot App built with the Java DSL can be versioned in Git, and it can be CD/GitOps friendly. With sufficient generalization to this App, it can also be reused by many different teams by overriding the defaults.

We put this for use in the product proper for or IT and Acceptance tests, which run on every upstream commit daily across multiple Kubernetes and Cloud Foundry installations.

Alternatively, all of the register, create, deploy, or launch stream/task commands can also be dumped in a text or a property file. Once when you have the file, the dataflow:>script --file command can help slurp in all the commands in each of the new environments — see docs.