2
votes

Is it possible to execute Bigquery's query using Google cloud data flow directly and fetch data, not reading data from table then putting conditions?

For example, PCollections res=p.apply(BigqueryIO.execute("Select col1,col2 from publicdata:samples.shakeseare where ...."))

Instead of reinventing using iterative method what Bigquery queries already implemented, we can use the same directly.

Thanks and Regards

Ajay K N

1

1 Answers

1
votes

BigQueryIO currently only supports reading from a Table and not a Query or View (FAQ).

One way to work around this is in your main program to create a BigQuery permanent table by issuing a query before you run your Dataflow job. After, your job runs you could delete the table.