Streaming pipeline publish to pubsub after write step completes

Question

I have a use case where I have a Dataflow job running in streaming mode with an hourly fixed window.

When the pipeline runs for a given window, we calculate some data and write it to a data source. What I want to do next is publish some message to PubSub once the write is complete - how might I go about making sure that the write step is complete before writing to PubSub?

If the pipeline was executed in batch mode I know I could execute it in a blocking fashion as suggested here, but the tricky part is that this constantly running in streaming mode.

Where do you write? If in Storage, you could use Pub/Sub notifications for Storage — Lefteris S
Good point, thanks for clarifying - in this case I am writing to BigQuery, but the same issue would apply to any database. — gilmatic

Raghu Angadi Raghu Angadi · Accepted Answer · 2018-06-22T18:20:13

3

votes

Wait.on() transform is designed for this use case. See documentation for usage example.

Streaming pipeline publish to pubsub after write step completes

1 Answers