GCP - Insert/Load Data to BigQuery from REST API using Google Cloud Function with Python

Question

Is it possible to directly insert/stream data from a rest api with json output to bigquery in gcp using cloud function?

Doing some research but it always ask to be put in GCS bucket then create the cloud function to load the data to bigquery.

thanks in advance.

Answer 1

Yes you can load it with no need of passing through GCS. Using this tutorial as a guide (which uses JSON files) brings us to this Source Code (used by the Cloud Function in the tutorial), as you can see all it does is load the bucket file into a blob and then to a json :

row = json.loads(blob.download_as_string())

After that is just a matter of using the insert_rows_json method :

BQ.insert_rows_json(table, json_rows=[row],
                             row_ids=[file_name],
                             retry=retry.Retry(deadline=30))

So, you can just send from your API the requests with a POST to a HTTP Cloud Function and that function use the above code to load the json fo BigQuery.