Google Cloud Dataflow - Apache Beam - Pipeline Shutdown Hook

Question

Wondering if there is some kind of "hook" to place a piece of code that would be executed when apache beam pipeline is being shutdown (for whatever reason - crash, cancel)

I need to delete a subscription on pubsub topic each time Dataflow is stopped.

Answer 1

Apache Beam is not naturally suited for this sort of flow. For this you may want to look at an orchestration engine, such as Apache Airflow.

With Airflow you should be able to schedule any sort of script to run after a Beam pipeline finishes/fails/is cancelled, etc. Take a look at it!

Answer 2

There are some examples of waiting for pipelines to finish and indeed managing Pubsub topics/subscriptions in a ExampleUtils class in the examples folder in the apache/beam repository here. See if there is anything you can use in the waitUntilFinish and tearDown methods.

This is java code - not sure if that is the language you use.

(In the long run @Pablo's suggestion to separate this further from the pipeline code may be best - perhaps depends on your exact goal here.)