GCP Dataflow Job Deployment

Question

I am trying to automate CI/CD of Classic Template.

Created and Staged template on GCS following the documentation

On code changes (Bug fixes etc), I intend to drain the existing job and create a new job with the same name.

To drain existing job, I need JOB_ID, but I have only JOB_NAME which I used during the creation of the job.

The only way i see is to use list command and fetch the active jobs, process the output to extract the job id to use it in the drain command. It seems to be quite a roundabout way. Isn't there a way to Drain a job with Job_Name or at least get JOB_ID from JOB_NAME.

Answer 1

When you use the gcloud dataflow jobs run command to create the job, the response from running this command should return the JOB_ID in the following way (e.g. if you create a batch job):

    id: 2016-10-11_17_10_59-1234530157620696789
    projectId: YOUR_PROJECT_ID
    type: JOB_TYPE_BATCH

That and using the gcloud dataflow jobs list as you mention will be the straightforward way to associate a JOB_NAME and a JOB_ID using automation. The way to achieve this with a Python script is described within this other post in the community.

Answer 2

GCP provides REST API to update dataflow job. No need of explicitly draining the existing the job and creating a new job.

You can do it via Python Code too. Refer to my GIST for python code.