SCDF Local data flow server

Question

To evaluate SCDF we have been experimenting with local server mode and the documentation states this is for development only not sure what that means. Question : But for production deployments does SCDF only come with kubernetes, apache yarn or cloud foundry target runtime. We dont have any of these current runtimes. We deploy microservices using ansible, and have indigenous scripts to manage our infrastructure and microservices.

For example is it not possible to alter the docker-compose yml to point to clustered kafka but not sure if dataflow server it self needs to be clustered or distributed (to mitigate single point of failure) may be have more than one data flow server processes running behind a load balancer.

Please advise, if local data server mode is applicable for production loads. Or what does cloud foundry or kubernetes version of SCDF provide extra in comparison to local data server.

Answer 1

The platform (i.e., cf, k8s, ..) is a requirement to run the stream/task apps reliably and to automatically also provide resiliency when the Apps crash for any reason. The platform will bring it back up without any manual intervention. Apart from that, a platform could provide end-to-end security, logging, metrics, autoscaling capabilities, job-scheduler, and among other things. SCDF relies on and delegates it to the platform to take advantage of these value-adds.

For example, to schedule a Task/Batch-Job with a cron-expression in PCF, you'd use PCF Scheduler and we have native integration for it from SCDF's Dashboard. This won't be available in the Local server.

It appears you have a homegrown platform-like infrastructure in place to provide HA, FT, and service-level guarantees to the Apps. Over the period, you may want to question the R&D efforts on that, but that's a different topic altogether. It is about choosing a platform vs. DIY, which I will skip in this response.

Back to your question now.

You could avoid SPOF for the Local-server by having multiple instances and a load-balancer in front. You just have to keep in mind that there's no automation to bring those instances back up and running on failures. Perhaps you may have something in your DIY-platform to address this, too.

And, yes, you can customize the docker-compose template to plug anything you want. In the end, it is going to help provision the relevant components to then build stream/task definitions using it.