How to execute Google Data Fusion Pipeline from a event based Cloud Function

Question

Is there any way to run a Google Data Fusion pipeline from a Cloud Function (preferably python based)?

The core requirement is, an event based Cloud function will execute whenever a new file arrives within a GCS bucket. The Cloud Function, in turn needs to call a Data Fusion pipeline which loads the GCS bucket file into BigQuery.

To execute the Cloud Function, we can use following:

gcloud functions deploy hello_gcs_generic --runtime python37 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.finalize

Answer 1

This is currently not possible. However, by GA, we will expose REST APIs to run data pipelines. You can then use the REST APIs to trigger pipelines from Cloud Functions.

Answer 2

You can use API from Cloud functions to start a pipeline.

https://cloud.google.com/data-fusion/docs/reference/cdap-reference#start_a_batch_pipeline