Spring Cloud Dataflow server - deployed streams ended up in unknown state

Question

Currently I am using spring-cloud-dataflow-local server with postgres database. I am using spring-cloud-stream applications for importing data from different sources, transform the data and finally store data in the postgres database.

I have deployed and started spring-cloud-dataflow local server on ec2 instance using Rancher. Then I have created and deployed streams using spring-cloud-datashell. Everything seems to be fine, we were planning to use same approach for all other data import processes.

Due to some reason rancher has restarted dataflow server, and my deployed streams ended up in unknown state. I have to log into dataflow server dashboard to restore the state of the streams by redeploying the streams manually.

My questions are Why spring-cloud-dataflow-local server does not remember deployment status of my stream once dataflow server has restarted and why it does not automatically recover. And how to avoid this behaviour. Any suggestions would be appreciated.

Thanks for your help.

Answer 1

Local server, if you are running with defaults will use an embedded database, on restarts it will loose it's state.

Also note that local-server is intended for development only, running a local server and deploying streams on a production server is a risky situation, as all process will be running on the same server, so you don't get any HA benefits, and they all compete for resources.

I'd recommend you take a look in the cloudfoundry, kubernetes or Yarn deployers that are offered.