Google Cloud Dataflow Pipeline Setup

Question

I currently have a pipeline setup and working on Streaming PubSub messages. Since this is a Streaming pipeline, it is going to run indefinitely. I would like to know two things.

• Is there any scenario in which such a running pipeline might be stopped automatically without user-intervention? Like if Pubsub runs out of messages for a really long time etc... If that is the case, then I might need to have some monitoring over the pipeline's lifecycle.

• I wish to maintain a unique pipeline - so each time I wish to make a change, I would like to kill the older one and then start a new one. I guess the name of the pipeline uniquely identifies it and so, I cant start a new one when the old one with the same name is running. But is there a way I can kill-start the pipeline with the same name?

  Both these questions revolve around maintaining pipelines with specific tasks. If there is any resource around the best practices for that, it would be helpful as well.

Answer 1

To your first question, there is no automated mechanism or TTL for streaming pipelines -- they will run until cancelled by the user.

For pipeline maintenance, Dataflow has an "update" feature that can be used to update a pipeline (of the same name) with new code: https://cloud.google.com/dataflow/pipelines/updating-a-pipeline