I have a streaming dataflow pipeline job which reads messages from a given pub-sub topic. I understand there is an auto-ack once the bundles are committed. How to make the pipeline stop where there are no more messages to consume?
Edit - I have a scenario where I need to drain off incorrect messages sent to the topic. Thus this would a one time job. My application sends 1MM messages only once a day (fixed time) to that topic.
Why would you want to stop the streaming pipeline, concerned about the being charged when the pipeline is doing nothing? If this is your concern then you should not be concerned at all since you will only be charged for the resources you use e.g CPU hour, Memory, Disk Storage, etc. please see pricing details here
Since your source is unbounded (e.g PubSub) then there's no way you could tell that there will be no more incoming data in the future.
