Trigger a dataflow job deployed through Cloud Run on object creation in GCP Storage Bucket

0
votes

I have created a dataflow pipeline which read a file from GCS bucket and process it. It is working when I execute the job from my local.

I deployed the dataflow job in Cloud Run with trigger on storage.object.create.

But when I upload any file in GCS bucket, no trigger message shows in the log or dataflow job not executed.

Trigger config

Ingress:Allow traffic
Authentication:Allow authentication
Event source:Cloud Storage
Event type:google.cloud.audit.log.v1.written
Create time:2021-02-12 (16:05:25)
Receive events from:All regions (global)
Service URL path:/
Service account:[email protected]
Service name:storage.googleapis.com
Method name:storage.objects.create

What am I missing here? Please suggest.

1
google-cloud-storagegoogle-cloud-dataflowgoogle-cloud-rundataflow
You can check this article if you haven't medium.com/@aishwarya.gupta3/… - Marco Massetti
Hi OP, please provide more details. What's your trigger config? What's the region/location of your bucket? How can we reproduce your issue? Also, note that event arc triggers can take up to 10 minutes to work after creation. Can you trigger the event again? - Donnald Cucharo
I didn't really understand the relation between dataflow and cloud run. Do you run dataflow direct runner inside Cloud Run? - guillaume blaquiere
@guillaumeblaquiere I am deploying my dataflowrunner pipeline job inside cloud run after making an image. I need to trigger this job when a file in uploaded in the gcs bucket. - Soumik Das
@DonnaldCucharo I waited for 10 mins after the trigger creation, but trigger does not working for gcs object creation. Trigger config updated in the question description. - Soumik Das

1 Answers

1
votes

The reason why your Cloud Run service isn't triggered is because there might be no audit logs written whenever an object is created/uploaded to your bucket. An Eventarc trigger is initiated whenever an event is written on Audit logs and by default, Cloud Storage is disabled:

The solution is to enable Audit Logs for Cloud Storage. It can be done two ways:

  1. Enable it on the first time you create an Eventarc trigger. enter image description here
  2. Or go to IAM & Admin > Audit logs and make sure that all fields are checked for Cloud Storage: enter image description here

As a reference, Audit logs can be seen on Home > Activity, here's an example:

enter image description here