Process streaming data from BigQuery table using Dataflow

Question

Like the example shown in https://github.com/GoogleCloudPlatform/DataflowJavaSDK-examples/blob/master/src/main/java/com/google/cloud/dataflow/examples/cookbook/TriggerExample.java

There is a BigQuery table where new data gets appended eveny 15 minutes. There is a Timestamp column in table. Is it possible to perform streaming analysis by fixedWindow time-based trigger from data being added to that BigQuery table? similar to the above example which uses pub/sub?

Hi, That's a bit of a broad question to answer, so it may not be a good fit for Stack... In any case, there's no triggers on BigQuery, so you would need to trigger it with a recurring cronjob I'd say. — Patrice

Dan Halperin Dan Halperin · Accepted Answer · 2015-10-02T04:03:26

Streaming data out of BigQuery is tricky -- unlike PubSub, BigQuery does not have a "subscribe to notifications" API. Is there a way you can stream upstream from BigQuery -- i.e., can you stream from whoever is pushing the 15-minute updates?

Process streaming data from BigQuery table using Dataflow

1 Answers