scheduled export from Google BigQuery to Google Cloud Storage

6
votes

I'm new to Google Cloud and would like to know best use cases on how to schedule queries and export them to Google Cloud Storage. I've seen documentations on how to manually export data but couldn't find anything specific on doing it in an automated way. Is there any best way on how to approach this ?

Thanks

1
google-bigqueryscheduled-tasksgoogle-cloud-storage
cloud.google.com/solutions/… – Elliott Brossard
In this repository (still under construction) I just implemented that using AppEngine Standard, main files are main.py, worker.py, they are setup by the yaml files ("app", "queue", "cron" and "worker"). Everyday at 10am I have a new file being exported from BQ to GCS which feeds some ML algorithms running everyday as well. – Willian Fuks
You can also use Apps Script. I wrote a blog - shinesolutions.com/2017/11/01/… – Graham Polley
Guys, thanks a lot for the responses. I managed using a Python script with export_data_to_gcs which exports a table to GCS - now I need to find how to export a query to GCS using python. any thoughts ? – RoS
There are some other StackOverflow questions (or see the BigQuery documentation) about using a destination table with Python. Run the query, then export the resulting table to GCS. – Elliott Brossard

1 Answers

4
votes
  1. Implement your table export function [1] using Node.js, python or Go. These languages are supported by Cloud Functions and BigQuery.
  2. Insert the above function in Cloud Function [2] service which is an event driven serverless compute platform.
  3. Trigger the above function using Cloud Scheduler [3] using a schedule interval of your choice. The schedule interval can be specified using a cron expression. The scheduler will trigger the function via a REST call on the function´s endpoint.
  4. Verify the success of the above operation by visiting your bucket and ensuring that the table(s) has been successfully exported.