Loading data into BigQuery from Cloud Storage using Cloud Function (Alternative to functions?)

1
votes

I'm currently loading data from a csv file in Cloud Storage to BigQuery using a Cloud Function.

However, Cloud Functions have a timeout limit and I expect these files to get big enough where it will cause issues.

In addition, inserting into big query is limited to 10mb/insert_rows.

What else can I use instead of Cloud Functions to get this working? Need it to be triggered upon file drop into Cloud Storage.

1
google-cloud-platformgoogle-bigquerygoogle-cloud-storagegoogle-cloud-functionsetl

1 Answers

2
votes

If you are inserting files from a .CSV use a batch load job. Asking BigQuery to load a file is a sub-second job, and BigQuery will proceed loading them in an async mode.

These loads are free, and your cloud function won't need to wait for it to finish.

You will be able to set up a notification that calls the cloud function each time a new file is dropped into your GCS bucket.