Google cloud build python apache beam data flow yaml file

0
votes

I am trying to deploy an apache beam Data Flow pipeline built-in python in google cloud build. I don't find any specific details about constructing the cloud build.YAML file.

I found a link dataflow-ci-cd-with-cloudbuild, but this seems JAVA based, tried this too but did not work as my starting point is main.py

enter image description here

1
google-cloud-dataflowapache-beamgoogle-cloud-buildcloudbuild.yaml
Do you want to run your dataflow from CloudBuild? Or to run a beam pipeline in Cloud Build runtime? Can you also share how you run today the dataflow, without cloud build? - guillaume blaquiere
The intent is to automate template creation. The above steps are taken from medium.com/@zhongchen/…. Seems need own docker image to build this github.com/GoogleCloudPlatform/cloud-builders-community/tree/… - Vivek Ranjan
How do you do today without Cloud Build? - guillaume blaquiere
It's done with cloud build using the yaml file from here medium.com/@zhongchen/… - Vivek Ranjan

1 Answers

0
votes

It requires a container registry. Step to build & deploy is explained in the below link

Github link