BigQuery Schedule Query Notification

Question

I want to get the result of my schedule query as soon as it completes, Big Query scheduler API provide pub-sub notification but the message doesn't contain anything about the result.

There is a consumer service which is waiting for the schedule query result we want to notify the consumer service once we have result ready and the service can start consuming it, How we can achieve this ?

The result is stored in the table but there are multiple schedule queries writing the result to the same table, so consumer service, can't just read my table as we need have some offset. So any help regarding this ?

I am a little confused with your question, do you want to notify the external service to start reading the data from big query table as soon as your schedule query that load data into the table is completed? I quite didnt understand "The result is stored in the table but there are multiple schedule queries writing the result to the same table, so consumer service, can't just read my table as we need have some offset", Do you want the service to read only the data that is loaded by your schedule query?? Please clarify. — radhika sharma

guillaume blaquiere guillaume blaquiere · Accepted Answer · 2021-08-10T12:55:36

You have 2 solutions:

With scheduled queries, natively, if a consumer expect the result of a specific scheduled queries (and don't care about the others), create a topic dedicated to this scheduled query and plug the consumer on this topic.
The other solution is to use Cloud Workflows instead of scheduled queries. With this design, Workflow performs a query with the dedicated connectors, and when the query is over, Workflow calls the consumer (or publish a message in PubSub with the correct attribute to let the consumers filter on the message they want.)

BigQuery Schedule Query Notification

1 Answers