Using Google Cloud Platform for scheduled recurring API data pull that then loads the data to BigQuery

0
votes

I need to setup a scheduled daily job that pulls data using a REST API call and then inserts that data into BigQuery.

I traditionally have done these types of tasks using Node.js running on Heroku. My current boss wants me to achieve this using the Google Cloud Platform.

What are some ways to achieve this using Google Cloud Platform?

1
google-bigquerygoogle-cloud-platform

1 Answers

2
votes

A few options on GCP:

  1. Spin up a GCE instance and use a cron (a little old school, but will work).
  2. Use Google App Engine and schedule your job(s) that way.

Unfortunately, Google Cloud Functions don't yet support schedulers. Otherwise, that would be perfect.