Google Cloud Dataflow for streaming, when will dataflow automatically shut down and clean up the VM instances?

Question

Based on the document, https://cloud.google.com/dataflow/docs/resources/faq

"However, once your job either completes or fails, the Cloud Dataflow service will automatically shut down and clean up the VM instances."

This is easy to understand for Batch. But for streaming, the pub/sub message will keep on coming, how dataflow handle this? it is not efficient to keep on shutting down and cleaning up the VM instances. But also it is not cost effective that keep the VM instances running all the time.

Thanks

Answer 1

With a streaming Dataflow job, the VMs will continue running until you end your job. If you don't have a constant flow of messages that are coming through pubsub you might want to look into cron jobs for batch Dataflow (https://cloud.google.com/blog/products/gcp/scheduling-dataflow-pipelines-using-app-engine-cron-service-or-cloud-functions) or use smaller worker nodes for your streaming job to help reduce cost.